fix(tests): make e2e backend model downloads resumable and stall-based#10766
Merged
Conversation
The vibevoice transcription e2e hangs until the go test timeout when the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to fit), and curl's --retry restarts from byte zero, so no attempt ever makes forward progress. This killed the job twice on PR #10764 and previously forced skipping it on release tags (#10567). Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s over --speed-time 120s) and resume from the bytes already on disk with -C -, retrying from Go because curl does not re-evaluate the resume offset on its internal retries. Resume against the HF Xet CDN was verified by killing a transfer mid-flight and confirming the next invocation appended (114 MB -> 235 MB, GGUF magic intact). Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default 30m) and raise it to 120m for the vibevoice transcription wrapper: a 10 GB download plus 25 specs does not fit in 30m even on a good day, and the job-level GHA timeout there is already 150m. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-fable-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
tests-vibevoice-cpp-grpc-transcriptionhangs until the innergo test -timeout 30mfires whenever the HF CDN has a slow day, and it just failed twice in a row on #10764 (the goroutine dump shows the suite parked insidedownloadFilefor the full 30 minutes). The same root cause previously forced skipping the job on release tags (#10567).Why it can never recover today:
x-linked-size: 10392063296)downloadFilecaps every curl attempt at--max-time 600, which needs a sustained ~17 MB/s to fit--retryrestarts from byte zero, so 6 capped attempts make zero forward progress and just burn the suite timeoutFix
--speed-limit/--speed-time)-C -; the retry loop moves into Go because curl does not re-evaluate the resume offset on its internal retriesBACKEND_TEST_TIMEOUT, default unchanged at 30m) and raise it to 120m for the vibevoice transcription wrapper (a 10 GB pull + 25 specs doesn't fit 30m even on a good day; the job-level GHA timeout there is already 150m)Verification
go test -v -timeout 30m, the vibevoice wrapper renders-timeout 120mgolangci-lint run --new-from-merge-base=origin/master ./tests/e2e-backends/...: 0 issues🤖 Generated with Claude Code