REPL exit hangs until a resumed background job finishes (blocking reaper vs. Runtime::drop) — GH #126 part C, confirmed

[tobert]

Follow-up from #126 part C ("reported, unverified" in the original issue). This
is now confirmed reproduced with direct process/thread evidence. Deliberately
not fixed in this pass — the fix needs a small design decision plus a
nontrivial restructure, and deserves its own careful TDD cycle rather than a
guess bolted onto the #126 cleanup. Splitting it out per the original plan
(#126 was closed by #140 + #160 without this part).

Repro

1. Start the interactive REPL.
2. sh -c 'sleep 3600' (a long-running foreground job).
3. Ctrl-Z to stop it.
4. bg to resume it in the background.
5. exit.

Observed: the kaish process does not terminate — it stays alive,
blocked, for as long as the backgrounded job keeps running (I let it run 15s
in testing, versus the theorized full 3600s; nothing suggested it would ever
return early).

Evidence this is a real block, not just a busy/loaded machine

The test machine was under genuine heavy load while I did this (confirmed:
2GB free RAM, 42GB swap in use, load average 30-58 from ~8 concurrent build
swarms) — so I built in a control before trusting the result:

• Control (fresh PTY session, exit immediately, no backgrounded job):
  process exits in ~100ms, even under that same load.
• Repro case (steps above): kaish still had not exited after 15s
  (5s + a further bounded 10s poll), while its sleep 3600-family child was
  independently confirmed still alive and running (ps showed it in state
  S, executing the sleep syscall normally — i.e. nothing killed it, it just
  kept running exactly as a real backgrounded job should).
• /proc/<kaish_pid>/task/*/{wchan,status} during the hang showed:
  • one thread named tokio-runtime-w[orker] in state S, wchan=do_wait
    — do_wait is the kernel's internal name for the wait-queue function
    backing wait4/waitpid-family syscalls. This is bg's reaper thread,
    parked inside its blocking waitpid(pid, None) call.
  • the other threads (the REPL's driver thread(s)) parked on
    wchan=futex_do_wait — consistent with tokio::runtime::Runtime's
    default Drop blocking the dropping thread until outstanding
    blocking-pool work (block_in_place/spawn_blocking) completes.
  • (gdb -p <pid> -batch -ex "thread apply all bt" would have been more
    definitive but ptrace is blocked in this sandbox — Operation not permitted. The /proc evidence above is consistent and specific enough
    that I'm confident in it, but a full backtrace would be worth getting in
    an environment where ptrace is available.)

Mechanism (source-level)

• bg.rs's reaper (tokio::spawn(async move { tokio::task::block_in_place(|| loop { waitpid(pid, None) ... }) ... })) has its JoinHandle dropped
  immediately — never awaited. Dropping a JoinHandle does not cancel the
  task; it keeps running, tracked by the runtime, independent of the bg
  command that spawned it.
• fg.rs has a similar-looking block_in_place wait, but it's synchronously
  awaited as part of fg's own execution — by the time you're back at a
  prompt to type exit, that closure has already returned. This bug is
  specific to bg's detached, outlives-the-command reaper.
• kaish-repl's exit path (ProcessResult::Exit in lib.rs:process_line →
  run_with_overlay/run return Ok(()) → the Repl struct, and its
  runtime: tokio::runtime::Runtime field, drop normally at scope exit) never
  calls shutdown_background() or shutdown_timeout() anywhere — it's a bare
  Runtime::drop, which tokio documents as blocking the dropping thread until
  outstanding blocking-pool work finishes (that's precisely why
  shutdown_background() exists as an alternative).

Fix directions (not yet implemented — needs a decision + care)

Option 1 — make the reaper non-blocking (my lean): replace the
block_in_place + blocking waitpid(pid, None) loop with a plain async task
that polls waitpid(pid, WNOHANG) on a tokio::time::sleep interval. No
block_in_place, so nothing for Runtime::drop to wait on — a not-yet-
finished plain async task is simply dropped/cancelled at shutdown rather than
joined. Smallest-diff option; doesn't touch Repl's structure. Needs a sane
poll interval choice and a regression test.

Option 2 — bounded shutdown at REPL exit: call runtime.shutdown_timeout(D)
explicitly instead of relying on implicit Drop. Simpler in spirit, but
shutdown_timeout consumes self, so Repl.runtime: Runtime would need to
become Option<Runtime> (or similar) purely to support taking ownership at
exit — a wider, more mechanical diff across every self.runtime.block_on(...)
call site.

Design question either way: what should happen to a backgrounded job
when the shell exits? Real bash's default (huponexit off, the common
interactive default) is: the job is not signaled, becomes an orphan of
init, and keeps running independently. Both fix options above are compatible
with that (the job's OS process was never a child of any Rust object kaish
depends on to keep running — it keeps going either way); the only question is
whether kaish's reaper task gets to see it finish and log a completion
(probably not worth blocking exit for either way). Worth confirming this is
the desired behavior before implementing, rather than assuming.

Test plan for whoever picks this up

crates/kaish-repl/tests/pty_job_control.rs is the right home. A permanent
regression test doesn't need a full 3600s upper bound — background a
short-duration job (e.g. a couple seconds), send exit immediately, and
assert the kaish process itself exits promptly (well under the job's
duration), the way I did for this investigation's throwaway diagnostic tests
(not committed anywhere — this issue's repro steps + evidence above are the
record).