Blocking input shape
The canonical AoU short-variant source for a chromosome is a set of indexed BGZF VCF partitions. Each partition has the same multi-sample cohort header/order and owns disjoint genomic intervals. For the current chr22 source there are 404 such VCFs, all with an identical 535,662-sample header.
Genoray 3.1.0 does not currently have an entry point with these semantics:
from_vcf internally shards indexed reads from one physical VCF, but accepts one source path.
from_vcf_list requires exactly one sample per file and unions sample columns, so routing the AoU partitions through it would change the dataset meaning.
- Materializing a concatenated VCF outside Genoray defeats native streaming and is infeasible at this cohort scale.
This blocks nf-seqlab VCF -> SVAR2 on the native AoU source.
Proposed contract
Add an explicit position-sharded cohort constructor (name open to preference), e.g. SparseVar2.from_vcf_shards(...), that:
- accepts N indexed VCF/BCF paths plus each source partition's owned genomic intervals;
- validates identical sample names/order and compatible header/contig schemas before writing;
- rejects overlapping source ownership intervals;
- reads the native partitions concurrently under the existing
threads= budget;
- emits one globally position-sorted raw record stream into a single
ChunkAssembler, so normalization/left-alignment remains global across source boundaries; and
- preserves the existing
regions=, samples=, check_ref, INFO/FORMAT, signatures, atomic-write, and deterministic-output contracts.
The source intervals are important: they let a fixed worker pool safely parallelize indexed reads without assuming that file-name order or physical file boundaries are normalization boundaries. They are source metadata, not a newly created VCF shard layer.
Correctness gates
- Byte-identical parity with
from_vcf on a concatenated fixture.
- Indel left-aligning across a source boundary.
- Multi-sample header mismatch/order rejection.
- Overlap/gap and empty-partition behavior.
- Serial vs multi-thread output parity.
I am preparing a focused implementation because this is blocking the end-to-end AoU/nf-seqlab run. Please flag a preferred public API name/shape before merge if you want this folded into from_vcf instead.
Blocking input shape
The canonical AoU short-variant source for a chromosome is a set of indexed BGZF VCF partitions. Each partition has the same multi-sample cohort header/order and owns disjoint genomic intervals. For the current chr22 source there are 404 such VCFs, all with an identical 535,662-sample header.
Genoray 3.1.0 does not currently have an entry point with these semantics:
from_vcfinternally shards indexed reads from one physical VCF, but accepts one source path.from_vcf_listrequires exactly one sample per file and unions sample columns, so routing the AoU partitions through it would change the dataset meaning.This blocks nf-seqlab VCF -> SVAR2 on the native AoU source.
Proposed contract
Add an explicit position-sharded cohort constructor (name open to preference), e.g.
SparseVar2.from_vcf_shards(...), that:threads=budget;ChunkAssembler, so normalization/left-alignment remains global across source boundaries; andregions=,samples=,check_ref, INFO/FORMAT, signatures, atomic-write, and deterministic-output contracts.The source intervals are important: they let a fixed worker pool safely parallelize indexed reads without assuming that file-name order or physical file boundaries are normalization boundaries. They are source metadata, not a newly created VCF shard layer.
Correctness gates
from_vcfon a concatenated fixture.I am preparing a focused implementation because this is blocking the end-to-end AoU/nf-seqlab run. Please flag a preferred public API name/shape before merge if you want this folded into
from_vcfinstead.