ci(tests): shard the server suite across a 4-runner matrix - #717
Merged
Conversation
The PR gate spent 11:25 of its 12:40 in one Test step: the server suite (1398 fast-tier tests, ~2550 s summed) already scales ~perfectly with the 4 cores a single runner has (maxParallelThreads is pinned to 4 — #536), so the remaining parallelization is more runners, not more threads. scripts/partition-tests.py deterministically packs test CLASSES onto 4 shards (greedy by weight; scripts/test-shard-weights.json holds real trx seconds, new classes get a default) and emits per-shard dot-anchored FullyQualifiedName filters. Classes are parsed from source declarations, not file names (one file declares two classes today). Shard 1 additionally runs the 11 s Client.Tests suite and cross-checks `dotnet test --list-tests` output: every discovered test must match exactly one shard's filter, so a partition miss fails the build instead of silently never running a class. The duration guardrail runs per shard; trx artifacts are per-shard. Branch protection swaps its required check to the new `Tests passed` fan-in (matrix names carry the shard index and can't be required; the fan-in passes iff all shards succeeded or were skipped as docs-only with the `changes` helper green). Verified locally: all 4 shard filters together execute 461+376+237+324 = exactly the 1398 fast-tier tests, all green; partition verify covers all 1450 listed tests (incl. Slow). Closes #716 Co-Authored-By: Claude Fable 5 <[email protected]>
Four required checks, not five — CodeQL runs on PRs but was never in the required set (verified via the branch-protection API while swapping in `Tests passed` for #716). Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #716
What
The PR gate spent 11:25 of its 12:40 running the server suite on a single 4-vCPU runner — and the suite already scales ~perfectly in-process (2550 s summed ÷
maxParallelThreads: 4≈ the observed wall; the pin is deliberate, #536). So this fans the suite out over a 4-job matrix instead:scripts/partition-tests.py— deterministic class→shard packing (greedy by weight, heaviest first) + per-shard--filteremission. Tokens are dot-anchored (FullyQualifiedName~.Class.) because~is substring matching.scripts/test-shard-weights.json— real per-class trx seconds from run 30862522491; new classes get a 10 s default until the file is refreshed.dotnet test --list-testsoutput must match exactly one shard's filter — a nested class or naming collision fails the build loudly instead of silently never running. The parser doesn't key off the localized "Tests are available" header.Tests passedfan-in job — the single stable-name required check standing in for the matrix; passes iff all shards succeeded or were skipped as docs-only (with thechangeshelper itself green).Verification
verifymaps all 1450 listed tests (incl. Slow) to exactly one shard.Expected
PR gate ~12:40 → ~4:30; push-to-main full runs shrink proportionally.
⚠ Merge choreography
Branch protection must swap required check
Build + test (.NET, headless)→Tests passedat merge time (doing it before hurts nothing either — this PR's run produces bothTests passedand the shard checks). release.yml is untouched.🤖 Generated with Claude Code