Skip to content

Support position-sharded multi-sample VCF cohorts in SparseVar2 #125

Description

@bschilder

Blocking input shape

The canonical AoU short-variant source for a chromosome is a set of indexed BGZF VCF partitions. Each partition has the same multi-sample cohort header/order and owns disjoint genomic intervals. For the current chr22 source there are 404 such VCFs, all with an identical 535,662-sample header.

Genoray 3.1.0 does not currently have an entry point with these semantics:

  • from_vcf internally shards indexed reads from one physical VCF, but accepts one source path.
  • from_vcf_list requires exactly one sample per file and unions sample columns, so routing the AoU partitions through it would change the dataset meaning.
  • Materializing a concatenated VCF outside Genoray defeats native streaming and is infeasible at this cohort scale.

This blocks nf-seqlab VCF -> SVAR2 on the native AoU source.

Proposed contract

Add an explicit position-sharded cohort constructor (name open to preference), e.g. SparseVar2.from_vcf_shards(...), that:

  1. accepts N indexed VCF/BCF paths plus each source partition's owned genomic intervals;
  2. validates identical sample names/order and compatible header/contig schemas before writing;
  3. rejects overlapping source ownership intervals;
  4. reads the native partitions concurrently under the existing threads= budget;
  5. emits one globally position-sorted raw record stream into a single ChunkAssembler, so normalization/left-alignment remains global across source boundaries; and
  6. preserves the existing regions=, samples=, check_ref, INFO/FORMAT, signatures, atomic-write, and deterministic-output contracts.

The source intervals are important: they let a fixed worker pool safely parallelize indexed reads without assuming that file-name order or physical file boundaries are normalization boundaries. They are source metadata, not a newly created VCF shard layer.

Correctness gates

  • Byte-identical parity with from_vcf on a concatenated fixture.
  • Indel left-aligning across a source boundary.
  • Multi-sample header mismatch/order rejection.
  • Overlap/gap and empty-partition behavior.
  • Serial vs multi-thread output parity.

I am preparing a focused implementation because this is blocking the end-to-end AoU/nf-seqlab run. Please flag a preferred public API name/shape before merge if you want this folded into from_vcf instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions