from_vcf_list: memory accumulates across contigs (OOM on large cohorts); auto chunk_size ignores format_fields <43k inputs

[d-laub]

Summary

SparseVar2.from_vcf_list has two memory problems that make a whole-genome, large-cohort conversion OOM, even though each individual contig fits comfortably in RAM:

1. Auto chunk_size ignores format_fields staging below 43k inputs, so it defaults to 25_000 and OOMs on a large contig.
2. Memory is not released between contigs — peak RSS grows monotonically across the per-contig loop, so it eventually OOMs on a smaller contig than one it already finished.

Both reproduced on genoray 3.2.1, converting 16,007 single-sample BCFs (tumor WGS, no_reference=True, format_fields=[FormatField("VAF")], VAF is Number=A,Float).

Symptom 1 — auto chunk_size OOM

SparseVar2.from_vcf_list(
    "out.svar2", "single-sample-bcfs/",   # 16007 *.bcf
    no_reference=True,
    format_fields=[FormatField("VAF")],
    threads=32,                            # chunk_size left default (None)
)

OOM-killed at ~200 GB during the read phase of chr2 (read=97%, tx_dense=0/6 — actively reading, not stalled), after chr1 completed. The docs note the auto budget "does not bite until roughly 43k inputs, below which it returns the historical 25_000." At 16k samples with a FORMAT field, a 25_000-variant chunk stages 25000 * 16007 * 4 B ≈ 1.6 GB per in-flight format buffer, which (times the in-flight buffer count) blows past 200 GB. Passing chunk_size=2000 explicitly fixed the read-phase memory (bounded ~38 GB). Suggestion: fold the format_fields term into the auto chunk-size budget regardless of input count (the single-file from_vcf fixed default of 25_000 has the same latent issue but that path is a separate story), or at least warn when format_fields is set and chunk_size is left to the small-cohort default.

Symptom 2 — memory accumulates across contigs (the real blocker)

With chunk_size=2000 (read phase now bounded), a whole-genome run still OOMs — but the peak grows per contig:

Point	Peak RSS
chr1 finished	~119 GB
chr2 finished	(continued)
chr3 (OOM)	267 GB → killed at the 256 GB cap

chr3 (24.4M variants) is smaller than chr2 (29.7M), which had completed — so a single contig's footprint is not the problem; memory from earlier contigs is not being freed before the next one is processed. No fixed node size can hold all 25 contigs in one process if the trend continues.

Per-contig peak is modest and stable — a standalone single-contig run of chr1 used MaxRSS 119 GB and completed in 13 min. It's only the accumulation across the per-contig loop that OOMs.

Workaround (what I shipped)

Run from_vcf_list once per contig in a separate process (via regions="{contig}:1-{len}"), each writing parts/{contig}.svar2, then SparseVar2.concat the 25 disjoint per-contig stores. Each process exits and frees its memory, so peak stays ~119 GB. As a SLURM array (5 concurrent) the whole genome finished in ~1.5 h and concat verified available_fields == ['VAF'], 16007 samples, 25 contigs, sane VAF values.

Ask

1. Release per-contig working memory between contigs in the single-process whole-genome path (the main bug) — or, if the accumulation is inherent to the current architecture, document that large cohorts must be converted per-contig + concat.
2. Account for format_fields in the auto chunk_size budget below 43k inputs (or warn), so the default doesn't OOM on the first large contig.

Repro

from genoray import SparseVar2, FormatField
# whole-genome (reproduces symptom 2 with chunk_size set, symptom 1 without):
SparseVar2.from_vcf_list(
    "out.svar2", "single-sample-bcfs/",     # 16007 single-sample BCFs, one FORMAT field
    no_reference=True, format_fields=[FormatField("VAF")],
    threads=32, chunk_size=2000,
)
# watch RSS climb per contig (chr1 ~119G, ... OOM on a later, smaller contig).
# Per-contig (regions="chrN:1-LEN") in separate processes stays flat at ~119G each.

Data (maintainer has cluster access): /carter/shared/data/gdc/somatic/wgs_DR45/results/single-sample-bcfs/ (16007 *.bcf).