Batch alleles per netMHCpan process for large speedups - #218
Merged
Conversation
netMHCpan (and the other command-line predictors) previously ran one
process per (input file, allele). Each process reloads the prediction
network, so a run over many alleles reloaded it once per allele — the
dominant cost when scanning many peptides across an HLA panel.
Batch alleles into a single comma-separated `-a A1,A2,...` invocation so
the network loads once per group instead of once per allele. The output
groups identically afterward (results are keyed by peptide/offset/source),
and the parser already attributes each row to its own allele, so this is
behavior-preserving.
max_alleles_per_command controls grouping on BaseCommandlinePredictor:
- 1 (base default): one allele per command — historical behavior,
kept for predictors whose allele flag may not accept a list.
- "auto" (netMHCpan family default): batch alleles as far as possible
while keeping enough parallel processes (input files x groups) to use
the available cores, so many-allele runs speed up without pessimizing
small runs on otherwise-idle cores.
- None / <=0: all alleles in one command.
- k>1: at most k alleles per command.
Also expose max_peptides_per_file on the netMHCpan wrappers for tuning
file-level batching/parallelism.
Example: 2.38M peptides x 100 alleles on 8 cores drops from ~23,800
network-reloading processes to 238 (one per input file, all alleles
batched); a 5,000-peptide x 8-allele run still uses 8 processes, so no
regression on small workloads.
Tests: tests/test_allele_batching.py covers _allele_groups (including the
"auto" heuristic), _build_command comma-joining (per-allele prepare then
join), the predict_peptides wiring (process counts and -a arguments), and
the pan-family defaults — all without needing the binary. Multi-allele
output parsing is already covered by
test_multi_allele_netmhcpan41_skips_interblock_headers, and
test_netmhc_pan_multiple_alleles now asserts both alleles come back from
the single batched invocation (runs on the netMHCpan-integration CI job).
Claude-Session: https://claude.ai/code/session_01LZahFhBSCiehXTESCYQ7wG
test_netmhc_pan_batched_matches_per_allele runs the real netMHCpan binary both ways (all alleles in one `-a A,B,C` process vs one process per allele) and asserts identical (allele, peptide) -> score results. Runs on the netMHCpan-integration CI job. Verified locally on netMHCpan 4.2 (Darwin_arm64): identical scores, and a 20.6x speedup for 24 alleles x 40 peptides run serially (per-allele reloads the network 24 times; batched loads once). Claude-Session: https://claude.ai/code/session_01LZahFhBSCiehXTESCYQ7wG
The per-process startup that batching amortizes is only ~100ms for netMHCpan, and the marginal cost of an extra allele in a running process is ~15ms, so the benefit of batching saturates quickly (~85% of it by a group of ~20). Past that, bigger batches mostly add downside: one failing allele takes out the whole batch, one process holds more output in memory, and there are fewer processes to spread across cores. Bound the "auto" policy at AUTO_MAX_ALLELES_PER_COMMAND (20). "auto" still adapts group size down to keep the cores busy; the cap only bounds the upper end (many input files would otherwise let it pack every allele into one call). The explicit None/<=0 escape hatch remains unbounded, and an explicit integer is still honored as-is. Claude-Session: https://claude.ai/code/session_01LZahFhBSCiehXTESCYQ7wG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
netMHCpan (and the other command-line predictors) run one process per (input file, allele). Each process pays a fixed startup cost — spawn the tcsh wrapper + compiled binary and read the pseudo-sequence/weight tables — so a run over many alleles pays that cost once per allele. With small per-allele peptide counts the repeated process startup dominates, which is why large many-allele jobs see poor throughput.
Change
Batch alleles into a single comma-separated invocation,
-a A1,A2,..., so the fixed startup is paid once per group instead of once per allele.Behavior-preserving:
(peptide, offset, source_sequence_name), identical to before.MHCcolumn and attributes it per-row (test_multi_allele_netmhcpan41_skips_interblock_headerscovers exactly the multi-block output a batched-a A,Bcall produces, including the inter-blockDistance to training dataheaders).max_alleles_per_command(onBaseCommandlinePredictor)1(base default)"auto"(netMHCpan family default)input_files × groups) to occupy the cores, and cap the group atAUTO_MAX_ALLELES_PER_COMMAND(20)None/<=0k > 1kalleles per commandAlso exposes
max_peptides_per_fileon the netMHCpan wrappers.Why the cap (measured on netMHCpan 4.2c, Darwin_arm64)
The cost being amortized is modest, so batching saturates fast:
t(1 peptide, 1 allele, -BA)): ~0.08–0.10s — it is process/wrapper spawn + reading the pseudo-sequence tables, not a heavy neural-network load (-BAadds ~2ms).Amortized fixed-cost per allele is
~100ms/G + 15ms: G=1→115ms, G=10→25ms, G=20→20ms, G=∞→15ms. The knee is ~10–20, so"auto"caps at 20 — past that, bigger batches mostly add downside (one failing allele takes out the whole batch; one process holds more output in memory; fewer processes to spread across cores)."auto"also adapts down: with one input file on a 10-core box it makes ~10 small groups to keep the cores busy, not one big one.Verified end-to-end
-a B07:02,A01:01,C07:02,A02:01call produces the exact same 24 (allele, peptide) → affinity scores as four separate per-allele calls — zero mismatches. Codified astest_netmhc_pan_batched_matches_per_allele."auto"8 procs of 3 alleles 3.94s = ~2.7×, identical 48,000 predictions.Scope
Enabled by default only for the netMHCpan class-I family (2.8 / 3.0 / 4.0 / 4.1 / 4.2 and the auto-detecting
NetMHCpan()), whose-aaccepts a comma-separated list. Other command-line predictors keep one-allele-per-command.Tests
tests/test_allele_batching.py(binary-free):_allele_groupschunking, the"auto"heuristic and its cap,_build_commandper-allele-prepare-then-join,predict_peptideswiring (process counts + emitted-a), pan defaults.integration-netmhcCI job):test_netmhc_pan_batched_matches_per_allele,test_netmhc_pan_multiple_alleles(asserts both alleles return from one batched call),test_multi_allele_netmhcpan41_skips_interblock_headers.Version bumped 3.16.0 → 3.17.0.
https://claude.ai/code/session_01LZahFhBSCiehXTESCYQ7wG