[fix][evaluation] Guard against false experiment termination after zombie kill#586
Merged
Conversation
…mbie kill 并发度=1 时，若唯一的 Processing item 已成僵尸，ScanEvalItems 会因空闲 并发名额为 0 而跳过 scanToSubmit(不去捞 Queueing item)，随后 handleZombies 把僵尸置 Fail 移出 incomplete，导致离线模式 ExptEnd 看到 toSubmit==0 && incomplete==0 而误判实验完成、直接终止，遗漏仍在 Queueing 的 item。 对齐在线(Append)模式"终止前重扫、不信旧计数"的既有范式：在 exptBaseExec 新增 hasPendingAfterRescan，直接查 DB 是否仍有 Queueing/Processing 的 item (不受并发名额门控)，仅在准备收工的冷路径触发。Submit / FailRetry / RetryAll / RetryItems 四个离线模式的 ExptEnd 命中终止条件时先过该护栏， 仍有残留则继续 nextTick。Append 模式本身已重扫，不受影响。 补充单测覆盖护栏正/反路径。 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
补齐 zombie-kill 护栏 hasPendingAfterRescan 的单测覆盖： 除已有的 ExptSubmitExec 正例外，新增 FailRetry / RetryAll / RetryItems 三个 executor 的 RescanGuard 用例。断言在 toSubmit==incomplete==0 的误判场景下， 只要 DB 重扫仍有 Queueing/Processing item，ExptEnd 返回 nextTick=true 且不走 exptEnd。 RetryItems 用例额外覆盖 Lock + run log item 归属校验的前置路径。 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
tpfz
approved these changes
Jul 21, 2026
Tiny1028
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With concurrency=1, if the only Processing item becomes a zombie, ScanEvalItems skips scanToSubmit (no free concurrency slot) and never picks up Queueing items. handleZombies then marks the zombie as Failed and removes it from incomplete. In offline mode, ExptEnd sees toSubmit == 0 && incomplete == 0, wrongly concludes the experiment is done, and terminates it — leaving still-Queueing items behind.
Fix
Align offline mode with the online (Append) pattern of "rescan before ending, don't trust stale counts". Added hasPendingAfterRescan on exptBaseExec, which queries the DB directly for any Queueing/Processing items (not gated by concurrency slots), triggered only on the cold path right before finalizing. The four offline executors (Submit / FailRetry / RetryAll / RetryItems) now pass this guard before terminating; if anything remains, they continue with nextTick. Append mode already rescans and is unaffected.
Tests
Added unit tests covering both the positive and negative paths of the guard across all retry executors. In the misjudgment scenario (toSubmit == incomplete == 0), the tests assert that as long as the DB rescan still finds Queueing/Processing items, ExptEnd returns nextTick=true and does not finalize. The RetryItems case additionally covers the Lock + run-log item-ownership pre-checks.