Major update

[madscort]

Updates including:

• Automatic context creation, which deterministically removes the need for decodetables
• Added VEP modules with LOFTEE plugin
• Entire workflow is now both docker and singularity compatible
• Resource utilisation now has sensible defaults

[Copilot]

Pull request overview

This PR is a broad Nextflow pipeline refresh that introduces deterministic matrix context generation (reducing reliance on user-provided decode tables), adds new analysis capabilities (tabular compilation, VEP/LOFTEE annotation, NGSEP calling, variant rescue, reporting), and standardizes execution across Docker/Singularity with updated default resources and profiles.

Changes:

• Add automatic matrix context + decode generation, plus new pinpoint tabular outputs and optional HTML reporting/rescue.
• Add new modules/subworkflows (VEP+LOFTEE, NGSEP, rescue, mpileup, bgzip, unique VCF merge) and supporting environments/containers.
• Refactor container selection across modules for Docker vs Singularity and update configs/profiles/docs accordingly.

Reviewed changes

Copilot reviewed 111 out of 116 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
subworkflows/variant_rescue.nf	New rescue subworkflow wiring (VCF prep + mpileup + marbl).
subworkflows/pinpoint_vcf.nf	Rename/extend pinpoint VCF subworkflow interface.
subworkflows/pinpoint_tab.nf	New tabular pinpoint compilation + optional VEP annotation.
subworkflows/mapping.nf	Remove duplicate metrics step from mapping QC.
subworkflows/mapping_umi.nf	Remove duplicate metrics step from UMI mapping QC.
subworkflows/calling.nf	Add optional NGSEP caller.
subworkflows/annotation.nf	Gate snpEff inputs behind params.annotate.
README.md	Update CLI usage, inputs, outputs, and annotation docs.
nextflow.config	New defaults (haplotypes, annotate flags, new features) + include SSI profile.
next.ssi	Add SSI wrapper script for running the workflow.
next.pbs	Update PBS wrapper defaults/modules/cache location.
modules/vep.nf	New VEP annotation process (incl. LOFTEE plugin integration).
modules/vartable.nf	Update container selection for docker vs singularity.
modules/vartable_pins.nf	Update container selection for docker vs singularity.
modules/validate.nf	Update container selection for docker vs singularity.
modules/unique_vcf.nf	New process to merge/deduplicate VCFs for VEP input.
modules/umi_metrics.nf	Update container selection + invoke fgbio directly.
modules/ubam.nf	Update container selection for docker vs singularity.
modules/test.nf	Update container selection + fix optional: syntax.
modules/subset.nf	Update container selection + reference basename handling.
modules/snpsift_filter.nf	Switch to snpsift env/container + invoke SnpSift directly.
modules/snpsift_clinvar.nf	Switch to snpsift env/container + invoke SnpSift directly.
modules/snpeff.nf	Update container selection + invoke snpEff directly.
modules/rescue.nf	New marbl-based rescue process.
modules/report.nf	New report rendering process (Rmarkdown).
modules/pinpy.nf	Update container selection for docker vs singularity.
modules/pin_basic.nf	New tabular compilation process invoking pin_basic.py.
modules/pilot_pinpoint.nf	Update container selection for docker vs singularity.
modules/octopus.nf	Update container selection for docker vs singularity.
modules/normalise_vcf.nf	Update container selection + reference basename handling.
modules/ngsep.nf	New NGSEP variant calling process.
modules/multiqc.nf	Update container selection for docker vs singularity.
modules/mpileup.nf	New samtools mpileup process for rescue.
modules/mosdepth.nf	Update container selection for docker vs singularity.
modules/mergevcfs.nf	Update container selection for docker vs singularity.
modules/mergepins.nf	Update container selection for docker vs singularity.
modules/mergebam.nf	Update container selection + reference basename handling.
modules/merge_consensus.nf	Update container selection + dynamic JVM mem sizing.
modules/matrix_context.nf	New process to build matrix context + derived decode table.
modules/markduplicates.nf	Update container selection + dynamic JVM mem sizing.
modules/markduplicates_spark.nf	Update container selection + dynamic JVM mem sizing.
modules/markduplicates_fast.nf	Update container selection for docker vs singularity.
modules/lofreq.nf	Update container selection + reference basename handling.
modules/intervals.nf	Update container selection + reference basename handling.
modules/insert_size_metrics.nf	Switch to picard env/container + adjust outputs.
modules/index.nf	Update container selection for docker vs singularity.
modules/index_vcf.nf	Update container selection for docker vs singularity.
modules/indelqual.nf	Update container selection + reference basename handling.
modules/hs_metrics.nf	Update container selection + reference basename handling.
modules/haplotypecaller.nf	Update container selection + dynamic JVM mem + configurable haplotypes.
modules/haplotypecaller_truth.nf	Update container selection + dynamic JVM mem sizing.
modules/haplotypecaller_truth_joint.nf	Update container selection + dynamic JVM mem sizing.
modules/haplotypecaller_split.nf	Update container selection + dynamic JVM mem sizing.
modules/group_umi.nf	Update container selection + invoke fgbio directly.
modules/genotypegvcf.nf	Update container selection + dynamic JVM mem sizing.
modules/genotypegvcf_split.nf	Update container selection + dynamic JVM mem sizing.
modules/genomicsdb.nf	Update container selection + dynamic JVM mem sizing.
modules/gc_metrics.nf	Switch to picard env/container + reference basename handling.
modules/freebayes.nf	Update container selection + reference basename handling.
modules/flagstat.nf	Update container selection for docker vs singularity.
modules/filter.nf	Update container selection for docker vs singularity.
modules/filter_variants.nf	Update container selection for docker vs singularity.
modules/fastqc.nf	Update container selection for docker vs singularity.
modules/fastq.nf	Update container selection for docker vs singularity.
modules/extract_umi.nf	Update container selection + invoke fgbio directly.
modules/duplicate_metrics.nf	Remove deprecated duplicate metrics module.
modules/drop_pl.nf	Update container selection for docker vs singularity.
modules/downsample.nf	Update container selection for docker vs singularity.
modules/deepvariant.nf	Update container selection + use in-container binary path.
modules/crisp.nf	Reference basename handling update.
modules/cram.nf	Update container selection + reference basename handling.
modules/combinegvcfs.nf	Update container selection + reference basename handling.
modules/clean.nf	Update container selection for docker vs singularity.
modules/call_consensus.nf	Update container selection + invoke fgbio directly.
modules/bqsr.nf	Update container selection + reference basename handling.
modules/bgzip.nf	New bgzip+tabix process for normalized VCFs.
modules/bed_annotate.nf	Update container selection for docker vs singularity.
modules/bam.nf	Update container selection + reference basename handling.
modules/apply_bqsr.nf	Update container selection + reference basename handling.
modules/alignment.nf	Update container selection + reference basename handling.
modules/alignment_umi.nf	Add container selection + reference basename handling.
modules/alignment_metrics.nf	Update container selection + reference basename handling.
modules/addreadgroup.nf	Update container selection for docker vs singularity.
main.nf	Wire new context/pinpoint_tab/rescue/report flows + update pooltable/ref handling.
lib/Utils.groovy	Add helper for “main+index” channel creation.
envs/vep/environment.yaml	Add conda environment for VEP.
envs/vep/Dockerfile	Add Docker image for VEP + LOFTEE plugins.
envs/vep/build	Add build helper script for VEP image.
envs/snpsift/environment.yaml	Add dedicated conda env for SnpSift.
envs/rescue/environment.yaml	Add conda env for marbl rescue.
envs/rescue/Dockerfile	Add Docker image for marbl rescue env.
envs/rescue/build	Add build helper script for marbl image.
envs/r_env/environment.yaml	Expand R environment dependencies for reporting.
envs/r_env/Dockerfile	Add Docker image for report rendering env.
envs/r_env/build	Add build helper script for report image.
envs/pinpy/environment.yaml	Update bcftools + add samtools for pinpy env.
envs/pinpy/build	Remove unsupported windows platform build.
envs/picard/environment.yaml	Add picard conda env.
envs/ngsep/environment.yaml	Add conda env for NGSEP.
envs/ngsep/Dockerfile	Add Docker image for NGSEP jar wrapper.
envs/ngsep/build	Add build helper script for NGSEP image.
config.json	Update example config defaults (step, reference filename).
config_test.json	Update test config reference filename.
config_qc.json	Update QC config defaults and outputs.
conf/ssi.config	Add SSI execution profile.
conf/profiles.config	Update profile defaults and test reference filename.
conf/ngc.config	Update NGC profile params and execution settings.
conf/container.config	Restructure container params for docker/singularity + update base image.
bin/report.Rmd	Add HTML report template.
bin/pin_basic.py	Add tabular compilation script (pool + pinpoint variant tables).
bin/build_matrix.py	Add deterministic matrix context/decode builder.
assets/data/test_data/target_calling.bed	Add gene column to bed test data.
assets/data/test_data/pooltable.tsv	Update test pooltable format (dimension column).
.gitignore	Stop ignoring lib/, add .vscode.
.github/workflows/nextflow-test-container.yml	Minor whitespace cleanup.
.gitattributes	Enforce LF endings for workflow-related files.

Comments suppressed due to low confidence (1)

subworkflows/pinpoint_vcf.nf:31

• PINPOINT_VCF declares pooltable and matrix_context as inputs but they are not referenced anywhere in the workflow body. Unused inputs make the subworkflow interface harder to understand and can lead to unnecessary staging.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.