Mo edits

.github/workflows/pr-tests.yml

@@ -1,6 +1,7 @@
 name: Pull Request Tests
 
 on:
+  workflow_dispatch:
   pull_request:
     types:
       - opened
@@ -38,6 +39,20 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y samtools
 
+      - name: Set up Java for Nextflow
+        uses: actions/setup-java@v4
+        with:
+          distribution: "temurin"
+          java-version: "21"
+
+      - name: Set up Nextflow
+        env:
+          NXF_VER: "24.10.0"
+        run: |
+          curl -s https://get.nextflow.io | bash
+          sudo mv nextflow /usr/local/bin/
+          nextflow -version
+
       - name: Install ZipStrain with test dependencies
         working-directory: zipstrain
         run: |
@@ -51,7 +66,7 @@ jobs:
           OMP_NUM_THREADS: "1"
           MKL_NUM_THREADS: "1"
         run: |
-          python -m pytest tests -q \
+          python -m pytest tests -q -rs \
             --cov=src/zipstrain \
             --cov-report=xml \
             --cov-report=term-missing

.gitignore

@@ -208,3 +208,10 @@ marimo/_lsp/
 __marimo__/
 dockerfile/.DS_Store
 docs/.DS_Store
+
+# Local Nextflow cluster profiles
+conf.local.config
+
+# Nextflow runtime state
+.nextflow/
+.nextflow.log*

CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 message: "If you use ZipStrain, please cite the software using the metadata in this file."
 type: software
 title: "ZipStrain"
-version: "0.10.1"
+version: "1.0.0"
 license: "MIT"
 authors:
   - family-names: "Ghadermazi"

README.md

@@ -25,164 +25,59 @@
 
 Developed by Parsa Ghadermazi and the [Olm Lab](https://www.colorado.edu/lab/olm/), University of Colorado Boulder.
 
-## Overview
-
-<img src="docs/workflow.png" alt="ZipStrain workflow" width="520">
-
-ZipStrain provides a full workflow for strain-level metagenomic analysis:
+**Strain-level metagenomics in three commands:** ZipStrain maps reads, profiles them at single-nucleotide resolution, and compares samples by ANI to tell whether they share a strain.
 
-- Profile BAM files into nucleotide-resolution A/C/G/T count tables
-- Generate companion genome and gene coverage summaries during profiling
-- Compare samples with popANI, conANI, cosANI, IBS, and identical-gene metrics
-- Run pairwise comparisons with Polars or DuckDB engines
-- Scale out with resumable local or Slurm-backed task execution
-- Run end-to-end Nextflow pipelines from local reads or SRA accessions
-- Build reference bundles directly from Sylph abundance tables
-- Use the CLI for production workflows or the Python API for custom analysis
-
-## Installation
+<img src="docs/workflow.png" alt="ZipStrain workflow" width="520">
 
-Install from PyPI:
+## Install
 
 ```bash
-pip install zipstrain
-zipstrain --version
+# Conda (recommended — bundles samtools and the map aligners)
+conda create -n zipstrain -c conda-forge -c bioconda python=3.12 zipstrain bowtie2 samtools sylph
+conda activate zipstrain
 zipstrain test
 ```
 
-ZipStrain requires Python 3.12+.
-If you install with `pip`, install `samtools` separately.
-If you plan to use the matrix-store workflow from pip, install the optional matrix dependencies with `pip install "zipstrain[matrix]"`.
-
-Other supported installation paths:
-
-- Conda: `conda install -c conda-forge -c bioconda -c defaults zipstrain`
-- Docker: `docker run -it parsaghadermazi/zipstrain:<version> zipstrain test`
-- Apptainer: `apptainer run docker://parsaghadermazi/zipstrain:<version> zipstrain test`
-
-More details: [Installation Guide](https://OlmLab.github.io/ZipStrain/installation/)
+Prefer pip? `pip install zipstrain` (then install `samtools` separately). Full options: [Installation](https://olmlab.github.io/ZipStrain/installation/).
 
-## Command Layout
+## The three commands
 
-| Command | Purpose |
-| --- | --- |
-| `zipstrain profile` | Batch-profile multiple BAM files |
-| `zipstrain compare genomes` | Batch genome-level comparisons |
-| `zipstrain compare genes` | Batch gene-level comparisons |
-| `zipstrain utilities ...` | Single-sample tools, preparation helpers, format conversion, and database builders |
-| `zipstrain test` | Validate the local installation |
+### 1. `map` — reads → BAMs
 
-## Quick Start
-
-### 1. Prepare profiling assets
+Aligns reads to a reference and writes sorted BAMs plus a `samples.txt` ready for profiling. Omit `--reference-fasta` to let Sylph pick and build a reference automatically. Resumable if interrupted.
 
 ```bash
-zipstrain utilities prepare_profiling \
-  --reference-fasta reference_genomes.fna \
-  --gene-fasta reference_genomes_gene.fasta \
-  --stb-file reference_genomes.stb \
-  --output-dir profiling_assets
+zipstrain map -i reads.csv -o mapped \
+  --reference-fasta ref.fna --stb-file ref.stb
 ```
 
-`prepare_profiling` writes `null_model.parquet`, `genomes_bed_file.bed`, `gene_range_table.tsv`, and `genome_lengths.parquet` into `profiling_assets`.
+<img src="docs/assets/zipstrain_map.gif" alt="zipstrain map" width="640">
 
-### 2. Profile BAM files in batch
+### 2. `profile` — BAMs → nucleotide profiles
 
-`samples.csv` must contain `sample_name` and `bamfile` columns.
+Counts A/C/G/T at every reference position and writes per-genome stats (coverage, breadth, a present/absent call) and SNVs. Missing assets are auto-generated and cached.
 
 ```bash
-zipstrain profile \
-  --input-table samples.csv \
-  --stb-file reference_genomes.stb \
-  --null-model profiling_assets/null_model.parquet \
-  --gene-range-table profiling_assets/gene_range_table.tsv \
-  --bed-file profiling_assets/genomes_bed_file.bed \
-  --genome-length-file profiling_assets/genome_lengths.parquet \
-  --run-dir profile_run
+zipstrain profile -i mapped/samples.txt -f ref.fna -s ref.stb -r profiled
 ```
 
-If you also pass `--reference-fasta`, each profile gains `ref_base_bitmask` and the companion gene/genome stat tables gain `ref_ani`.
+<img src="docs/assets/zipstrain_profile.gif" alt="zipstrain profile" width="640">
 
-### 3. Build a profile DB
+### 3. `compare` — profiles → ANI
 
-Create a CSV with header `profile_name,profile_location`, then build the profile DB:
+Compares every pair of samples by popANI (near 100% ⇒ same strain). Point `--profile-db` at a CSV of `profile_name,profile_location`.
 
 ```bash
-zipstrain utilities build-profile-db \
-  --profile-db-csv profiles.csv \
-  --output-file profile_db.parquet
+zipstrain compare --profile-db profiles.csv -r compared
 ```
 
-### 4. Run genome comparisons
-
-Launch batched genome comparisons from the profile DB:
+<img src="docs/assets/zipstrain_compare.gif" alt="zipstrain compare" width="640">
 
-```bash
-zipstrain compare genomes \
-  --profile-db profile_db.parquet \
-  --scope all \
-  --stb-file reference_genomes.stb \
-  --run-dir genome_compare_run \
-  --calculate all
-```
-
-For ANI-only runs:
-
-```bash
-zipstrain compare genomes \
-  --profile-db profile_db.parquet \
-  --scope all \
-  --stb-file reference_genomes.stb \
-  --run-dir genome_compare_run \
-  --calculate ani
-```
-
-Set `--engine duckdb` to switch the genome-compare backend from the default Polars engine.
-
-Single-pair helpers and table builders live under `zipstrain utilities`.
-The full command reference is linked below.
-
-## Nextflow Workflows
-
-ZipStrain ships with Nextflow workflows for:
-
-- read mapping
-- BAM profiling
-- SRA-to-profile processing
-- genome comparisons
-- gene comparisons
-
-Example:
-
-```bash
-nextflow run zipstrain.nf \
-  --mode profile \
-  --input_table bams.csv \
-  --reference_genome reference_genomes.fna \
-  --gene_file reference_genomes_gene.fasta \
-  --stb reference_genomes.stb \
-  --output_dir out_profile \
-  -c conf.config \
-  -profile docker \
-  -resume
-```
-
-See: [Nextflow Pipeline Guide](https://OlmLab.github.io/ZipStrain/NextflowPipeline/)
+Every run writes a `zipstrain_run.log` so you can tell if it is running, finished, or crashed. Prefer a pipeline? A [Nextflow implementation](https://olmlab.github.io/ZipStrain/usermanual/#nextflow-implementation) runs all of this end to end.
 
 ## Documentation
 
-Full documentation is available at:
-
-**[OlmLab.github.io/ZipStrain](https://OlmLab.github.io/ZipStrain/)**
-
-| Page | What it covers |
-| --- | --- |
-| [Installation](https://OlmLab.github.io/ZipStrain/installation/) | PyPI, Conda, Docker, and Apptainer setup |
-| [CLI Reference](https://OlmLab.github.io/ZipStrain/cli/) | Commands, options, and workflow layout |
-| [Tutorial](https://OlmLab.github.io/ZipStrain/Tutorial/) | End-to-end usage examples |
-| [Nextflow Pipelines](https://OlmLab.github.io/ZipStrain/NextflowPipeline/) | Cluster-ready workflows |
-| [Build Genome DB](https://OlmLab.github.io/ZipStrain/GenomeDBFromSylph/) | Build reference bundles from Sylph output |
-| [Python API](https://OlmLab.github.io/ZipStrain/api/) | Programmatic usage |
+Full documentation available at [https://olmlab.github.io/ZipStrain/](https://olmlab.github.io/ZipStrain/)
 
 ## Citation
 

changelog.md

@@ -2,6 +2,56 @@
 
 Entries are brief by design and describe changes relative to the previous released version.
 
+## 1.0.0
+
+First stable release. This version consolidates ZipStrain around three top-level commands — `map`, `profile`, and `compare` — plus a `test` environment check and a `utilities` group of lower-level helpers. Compared with `0.11.3`:
+
+### Command-line interface
+
+- New streamlined top level: `zipstrain map | profile | compare | test`, with the previous lower-level helpers gathered under `zipstrain utilities`.
+- `compare` is now a single command: it defaults to genome-level comparison, takes `--compare-genes` for gene-level comparison, and selects the engine with `--method standard|matrix`. The old `zipstrain compare genomes` / `zipstrain compare genes` subcommands are gone.
+- `compare --profile-db` accepts a CSV of `profile_name,profile_location` directly, so a separate `build-profile-db` step is no longer required.
+- `compare` runs are resumable and extendable: re-running with the same `--run-dir` and a profiles table that adds samples computes only the new pairs. Both methods write a single `all_comparisons.parquet` (or `all_gene_comparisons.parquet`) at the top of the run directory.
+- `compare --calculate` now accepts fine-grained metric combos (for example `ani+ibs+identical_genes`) through the batched standard path, not just `all`/`ani`.
+- Utility cleanup: `generate-genome-pairs` is now `generate-sample-pair`; redundant coverage stats, BED, genome-length, and strain-heterogeneity helper commands were removed.
+- Grouped, sectioned `--help` output for the main commands.
+
+### Mapping (`zipstrain map`)
+
+- New command that turns a reads table into sorted, indexed BAMs ready for `profile`, and emits the reference FASTA + STB and a `samples.txt`.
+- Omitting `--reference-fasta` runs Sylph to auto-pick, download, and cache reference genomes from the reads, then maps against the built reference; a `reference_genomes_taxonomy.tsv` is written that `profile` auto-discovers.
+- Resumable: completed Sylph tables, the built reference, the Bowtie2 index, and each sorted+indexed BAM are reused on re-run (`--force` to rebuild). Optional `--predict-genes` (Prodigal) and `--non-competitive` mapping.
+
+### Profiling (`zipstrain profile`)
+
+- Profiling assets (null model, BED, genome lengths, gene range table, profiling contract) are auto-generated into `<run-dir>/profiling_assets` and cached, so a minimal run needs only `--input-table`, `--reference-fasta`, and `--stb-file`.
+- Reference-aware profiling: `--reference-fasta` adds `ref_base_bitmask` to profiles and `ref_ani` to gene/genome stats.
+- Per-sample SNV/SNP calling (`<sample>_SNVs.parquet`), with inStrain-style stats in genome/gene stats: `SNS_count`, `SNV_count`, `conANI_reference`, `ber`, `fug`, coverage summaries, and an automated `present`/`absent` `presence` call (BER/FUG thresholds).
+- Read/base filters aligned with inStrain and samtools: `--min-mapq`, `--min-baseq`, `--min-read-ani` (NM-tag based), and `--read-inclusion proper-pairs|paired|all-mapped` (default `paired`).
+- Optional `--genome-taxonomy` (auto-discovered from the Sylph route) adds a `genome_taxonomy` column; profile outputs are parquet-only.
+
+### Comparison outputs and metrics
+
+- Genome comparison tables keep the `0.11.2` metric schema, but the ANI column is now named `genome_ani`; the parquet header stores `zipstrain_compare_ani_method` so the metric can be interpreted as `popani`, `conani`, or `cosani_<threshold>`.
+- Matrix comparison is integrated as `compare --method matrix` (HDF5 matrix store, numpy/torch CPU/CUDA/MPS backends, optional sparse storage), while the low-level `utilities build-matrix-db` / `append-matrix-db` / `matrix-compare` / `matrix-compare-export` / `matrix-db-to-hdf5` remain available.
+- `utilities get-snp-reference` can emit either profile-like parquet or a site-only VCF.
+- `utilities parquet-to-csv` converts any parquet table to CSV explicitly.
+
+### Reliability and logging
+
+- Universal run logging: `map`, `profile`, and `compare` write a `zipstrain_run.log` and `zipstrain_status.json` so a run's state (running / finished / crashed) is always inspectable.
+- Profiling subprocess hardening: each task runs in its own process group with a timeout and group-kill on failure; fixed a profiling concurrency deadlock and now require same-scaffold read pairs by default.
+
+### Nextflow
+
+- Added a root `nextflow.config` so `nextflow run OlmLab/ZipStrain` works out of the box with Docker on a laptop; HPC Singularity/SLURM profiles live in `conf.config`.
+- Nextflow modes (`map_reads`, `from_sra_to_profile`, `profile`, `compare_genomes`, `compare_genes`) call the updated CLI and inherit run logging.
+
+### Documentation and packaging
+
+- Reorganized docs into a Quick Start, a comprehensive User Manual (CLI + Nextflow reference), an Expected Output reference, a Downstream Analyses guide, and updated Installation/Tutorial pages; added quickstart GIFs.
+- Fleshed-out `pyproject.toml` metadata with `matrix` (torch, h5py) and `test` extras.
+
 ## 0.11.3
 
 Compared with `0.11.2`: