Title: Scheduler Health Monitor: Diagnostics API and Built-in Watchdog
Environment
Background
(Split from the original exception handling improvements issue)
As discussed in #62, when a fiber blocks its scheduler thread (blocking I/O, infinite loop),
new fibers assigned to that thread are silently lost. This proposal provides a two-layer solution:
a data API for custom monitoring, and a built-in watchdog for out-of-the-box detection.
Proposed Solution
Layer 1: Scheduler Diagnostics API (data layer)
Expose internal scheduler state so users can inspect, log, and debug issues:
struct SchedulerInfo {
uint fibersCompleted;
uint fibersFailed;
uint pendingFibers;
MonoTime lastActivity; // last time this scheduler completed a fiber
}
SchedulerInfo[] getSchedulerInfo();
string dumpSchedulerState(); // formatted output for debuggingAdvanced users can build custom monitoring on top of this data:
// Custom health check in user code
auto infos = getSchedulerInfo();
foreach (i, ref info; infos) {
auto stalled = MonoTime.currTime - info.lastActivity;
if (stalled > 30.seconds && info.pendingFibers > 0) {
logError("Scheduler %d stalled for %s, %d fibers pending",
i, stalled, info.pendingFibers);
}
}Layer 2: Built-in Watchdog (convenience layer)
For most users, a built-in watchdog provides out-of-the-box detection of blocked scheduler threads:
struct WatchdogConfig {
Duration checkInterval = 5.seconds;
Duration stallTimeout = 30.seconds; // time without activity before alerting
void delegate(size_t schedulerId, Duration stalled, SchedulerInfo info) nothrow onStalled;
}
void enableWatchdog(WatchdogConfig config);
void disableWatchdog();Usage:
// Enable watchdog with custom callback
photon.enableWatchdog(WatchdogConfig(
checkInterval: 5.seconds,
stallTimeout: 30.seconds,
onStalled: (schedulerId, stalled, info) {
logError("Scheduler %d stalled for %s! pending=%d",
schedulerId, stalled, info.pendingFibers);
}
));
// Dump state on demand (e.g., HTTP admin endpoint)
void handleDebugRequest() {
writeln(photon.dumpSchedulerState());
}Expected Value
Detect blocked threads, diagnose deadlocks, and address the silent fiber loss issue (#62).
Title: Scheduler Health Monitor: Diagnostics API and Built-in Watchdog
Environment
~>0.19.1Background
(Split from the original exception handling improvements issue)
As discussed in #62, when a fiber blocks its scheduler thread (blocking I/O, infinite loop),
new fibers assigned to that thread are silently lost. This proposal provides a two-layer solution:
a data API for custom monitoring, and a built-in watchdog for out-of-the-box detection.
Proposed Solution
Layer 1: Scheduler Diagnostics API (data layer)
Expose internal scheduler state so users can inspect, log, and debug issues:
Advanced users can build custom monitoring on top of this data:
Layer 2: Built-in Watchdog (convenience layer)
For most users, a built-in watchdog provides out-of-the-box detection of blocked scheduler threads:
Usage:
Expected Value
Detect blocked threads, diagnose deadlocks, and address the silent fiber loss issue (#62).