InputMergeActor: fix deadlock

[woutdenolf]

An Ewoks ppf actor graph has an InputMergeActor in front of a PythonActor (see docs).

The deadlock can happen when the Ewoks task loops to itself so InputMergeActor triggers PythonActor which trigger the same InputMergeActor

InputMergeActor._execute (iteration N)      -- with self._lock:  ← lock acquired here, not yet released
  → _trigger_downstream
    → PythonActor.trigger → _execute
      → self.pool.apply_async(...)           (PythonActor)
        → ScalingPool.apply_async(...)
          → ThreadPool.apply_async(...)
            → future.add_done_callback(cb)   ← if worker already done, `cb` runs HERE, same thread, same stack

Of course this is very unlikely but could happen

        future = self._pool.submit(...)
        future.add_done_callback(cb)   # if worker already finished, `cb` runs in the current call stack

The fix is to make the lock in InputMergeActor re-entrant. Since this is the same call stack we don't have to worry these calls can happen concurrently, as opposed to calls from different upstream actors.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[woutdenolf]

This is the fix. The rest are unit tests.