diff --git a/.gitignore b/.gitignore index 9dbc8cee..64881f18 100644 --- a/.gitignore +++ b/.gitignore @@ -14,4 +14,5 @@ _site *_files/ site_libs -/.luarc.json \ No newline at end of file +/.luarc.json +**/*.quarto_ipynb diff --git a/CHANGELOG.md b/CHANGELOG.md index a6bd424a..06d392d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,21 +4,27 @@ * Add descriptions to all pages and add listings to index pages. -* Update documentation on creating components for developers. +* Update documentation on creating components for developers, including updated config examples reflecting Viash 0.9.x syntax (`argument_groups`, `engines`, `runners`). -* Update getting started page for developers +* Add guide for creating new pipelines, covering workflow structure, config layout, `main.nf` conventions, and testing. + +* Expand user guide: ingestion, processing, and downstream analysis pages now contain full content covering workflow inputs, outputs, and usage patterns. + +* Rewrite architecture page to provide a concise, diagram-light overview of how OpenPipelines fits into a single-cell project. + +* Update getting started page for developers. * Update project structure. -* Update information on running tests. +* Update information on running tests, including how to generate local test data without downloading from S3. -* Update "More information" pages +* Update "More information" pages. -* Write getting started page for user guide +* Write getting started page for user guide. -* Document how to run workflows +* Document how to run workflows. -* Document parameter lists +* Document parameter lists. # OpenPipelines.bio v0.12.1 diff --git a/contributing/creating_components.qmd b/contributing/creating_components.qmd index 5c83ce18..daa610bb 100644 --- a/contributing/creating_components.qmd +++ b/contributing/creating_components.qmd @@ -6,7 +6,7 @@ order: 20 ## A common file format -One of the core principals of OpenPipelines is to use [MuData](https://mudata.readthedocs.io/) as a common data format troughout the whole pipeline. See [the concepts page](/fundamentals/concepts.qmd#sec-common-file-format) for more information on openpipelines uses MuData to store single-cell data. +One of the core principals of OpenPipelines is to use [MuData](https://mudata.readthedocs.io/) as a common data format throughout the whole pipeline. See [the concepts page](/fundamentals/concepts.qmd#sec-common-file-format) for more information on openpipelines uses MuData to store single-cell data. ## Component location As discussed in [the project structure](project_structure.qmd#sec-project-structure), components in the repository are stored within `src`. Additionally, components are grouped into namespaces, according to a common functionality. An example of such a namespace is the dimensionality reduction namespace (`dimred`), of which the components `pca` and `umap` are members. This means that within `src`, the namespace folders can be found that stores the components that belong to these namespaces. @@ -27,31 +27,46 @@ A component consists of one or more scripts that provide the functionality of th ## The config ```yaml -functionality: - name: "my_component" - namespace: "my_namespace" - description: "My new custom component" - authors: - - __merge__: ../../authors/my_name.yaml - roles: [ author ] - arguments: - - name: "--output" - type: file - example: "output_file.h5mu" - description: "Location were the output file should be written to." - direction: "output" - resources: - - type: python_script - path: script.py -platforms: +name: "my_component" +namespace: "my_namespace" +description: "My new custom component" +authors: + - __merge__: ../../authors/my_name.yaml + roles: [ author ] + +argument_groups: + - name: "Inputs" + arguments: + - name: "--input" + type: file + required: true + description: "Input H5MU file." + example: "input.h5mu" + - name: "Outputs" + arguments: + - name: "--output" + type: file + direction: output + description: "Location where the output file should be written to." + example: "output.h5mu" + +resources: + - type: python_script + path: script.py + +engines: - type: docker image: python:3.11 setup: - type: python - packages: mudata~=0.2.3 + packages: + - mudata~=0.3 + +runners: - type: nextflow directives: label: [highcpu, midmem] + - type: executable ``` ### Basic information @@ -66,16 +81,16 @@ Basic information checklist: ### Arguments and argument groups -If you component requires arguments, they should be defined in `arguments` or `argument_groups`. Try tro group individual arguments into `argument_groups` when the number of arguments become too larg (10 or more as a rule of thumb). +If your component requires arguments, they should be defined in `argument_groups`. Try to group individual arguments into `argument_groups` when the number of arguments become too large (10 or more as a rule of thumb). Argument checklist: - Add a description and name - - Each argument should have the appropriate type. + - Each argument should have the appropriate type - Input and output files should be of type `file` instead of `string` and use the appropriate `direction:` - If possible: add an example - If the argument can accept multiple values, add `multiple: true` - - If the possible input for an argument is limited to certain set of values, use `choices:` + - If the possible input for an argument is limited to a certain set of values, use `choices:` ### (Test)resources @@ -84,49 +99,240 @@ Resources define files that are required for a component to perform its function There is a difference between defining `resources` and `test_resources`. While resources are required for a component to function, `test_resources` only need to be included when testing the component (with for example `viash test`) in addition to the regular resources. Having a look at the example above, resources are defined using the `resources:` property. It takes a list of multiple files or folders. In openpipelines, it was decided to not use a service like `git lfs` to include large resources into the repository. Instead, if large resources are required, there are two possibilities: + * Large resources required for _testing_ are to uploaded into an `s3` bucket that is synced automatically before running tests (both locally and on github). Please ping a maintainer when you open a PR and ask them to upload the files for you. - * Other large resources that are not needed for testing can be considered as input. This means that an argument of `type: file` needs to be created. The downside of this method is that viash is not able to natively support remote files f + * Other large resources that are not needed for testing can be considered as input. This means that an argument of `type: file` needs to be created. The downside of this method is that viash is not able to natively support remote files. Resources checklist: + - Script resources are located next to the config and added to the config with the correct type (`python_script`, `r_script`, ...) - - Small resources (<50MB) that are not scripts can also be checked in into the repo, next to the + - Small resources (<50MB) that are not scripts can also be checked into the repo, next to the `config.vsh.yaml`. ### The script file -TODO +The script file contains the actual implementation of the component. Viash injects the parsed CLI arguments into the script using a `## VIASH START` / `## VIASH END` marker block. During development, this block holds placeholder values so the script can be run directly in an IDE or REPL. At build time, Viash replaces the block with generated code that reads the actual CLI arguments. + +::: {.panel-tabset} + +# Python + +```python +## VIASH START +par = { + "input": "input.h5mu", + "output": "output.h5mu", + "modality": "rna", + "min_counts": 200, +} +meta = { + "resources_dir": ".", + "temp_dir": "/tmp", + "cpus": 1, + "memory_gb": 4, +} +## VIASH END + +import mudata as mu + +mdata = mu.read_h5mu(par["input"]) +adata = mdata.mod[par["modality"]] + +# ... component logic ... + +mdata.write_h5mu(par["output"]) +``` + +# R + +```r +## VIASH START +par <- list( + input = "input.h5mu", + output = "output.h5mu", + modality = "rna" +) +meta <- list( + resources_dir = ".", + temp_dir = "/tmp", + cpus = 1L, + memory_gb = 4L +) +## VIASH END + +library(MuData) + +mdata <- readH5MU(par$input) + +# ... component logic ... + +writeH5MU(mdata, par$output) +``` + +# Bash + +```bash +## VIASH START +par_input="input.h5mu" +par_output="output.h5mu" +par_modality="rna" +meta_cpus=1 +meta_memory_gb=4 +## VIASH END + +set -eo pipefail + +# unset boolean parameters before use +[[ "$par_some_flag" == "false" ]] && unset par_some_flag + +# build argument array +cmd_args=( + --input "$par_input" + --output "$par_output" + --modality "$par_modality" +) +[[ -n "$par_some_flag" ]] && cmd_args+=(--flag) + +mytool "${cmd_args[@]}" +``` + +::: + +Script checklist: + + - The `## VIASH START` / `## VIASH END` block is present with realistic placeholder values + - Use `meta_cpus` and `meta_memory_gb` for resource-aware parallelism — never hardcode thread counts + - Bash scripts: use `set -eo pipefail` after `## VIASH END` and build CLI arguments as arrays ### Author information -TODO +Author information is centralised in `src/authors/`. Each contributor has a YAML file with their profile, which is merged into individual component configs using Viash's `__merge__` property. This keeps author information consistent across all components without duplication. + +To add yourself as an author: + +1. Create `src/authors/your_name.yaml`: + +```yaml +name: Your Name +info: + links: + - type: github + url: https://github.com/your_handle + - type: orcid + url: https://orcid.org/0000-0000-0000-0000 +``` + +2. Reference it in your component's `config.vsh.yaml`: + +```yaml +authors: + - __merge__: ../../authors/your_name.yaml + roles: [ author ] +``` + +Valid roles are `author` and `maintainer`. The `maintainer` role indicates who is responsible for keeping the component up to date. ## Adding dependencies -TODO +Dependencies are declared in the `engines` section of `config.vsh.yaml`. Viash uses this information to build the Docker image for the component. Always prefer installing packages through a package manager over building from source. + +::: {.panel-tabset} + +# Python (pip) + +```yaml +engines: + - type: docker + image: python:3.11 + setup: + - type: python + packages: + - mudata~=0.3 + - scanpy>=1.9 + - numpy +``` + +# Python (conda / BioContainers) + +BioContainers images from `quay.io/biocontainers` are preferred for bioinformatics tools because they pin exact versions including the build string (e.g. `tool:2.0.3--h5b5514e_1`), improving reproducibility: + +```yaml +engines: + - type: docker + image: quay.io/biocontainers/scanpy:1.9.3--pyhdfd78af_0 + setup: + - type: python + packages: + - mudata~=0.3 +``` + +# R + +```yaml +engines: + - type: docker + image: eddelbuettel/r2u:22.04 + setup: + - type: r + cran: + - MuData + - Seurat + bioc: + - SingleR + - scran +``` + +# System packages (apt) + +```yaml +engines: + - type: docker + image: ubuntu:22.04 + setup: + - type: apt + packages: + - samtools + - libhdf5-dev + - type: python + packages: + - mudata +``` + +::: + +After editing the setup, rebuild the Docker image to verify it installs correctly: + +```bash +viash build src/my_namespace/my_component/config.vsh.yaml \ + --engine docker --setup cb +``` + +The `cb` (cachedbuild) flag reuses cached layers where possible. ## Building components from their source -When running or [testing individual components](#running-component-unittests), it is not necessary to execute an extra command to run the build step, `viash test` and `viash run` will build the component on the fly. However, before integrating components into a pipeline, you will need to build the components. More specifically, openpipelines uses Nextflow to combine components into pipelines, so we need to have at least the components build for `nextflow` platform as target. The easiest method to build the components is to use: +When running or [testing individual components](running_tests.qmd), it is not necessary to execute an extra command to run the build step, `viash test` and `viash run` will build the component on the fly. However, before integrating components into a pipeline, you will need to build the components. More specifically, openpipelines uses Nextflow to combine components into pipelines, so we need to have at least the components build for `nextflow` platform as target. The easiest method to build the components is to use: ```bash -viash ns build --parallel --setup cachedbuild +viash ns build --setup cb ``` -After using `viash ns build`, the target folder will be populated with three subfolders, corresponding to the build platforms that viash supports: `native`, `docker` and `nextflow`. +After using `viash ns build`, the `target/` folder is populated with build outputs for each declared engine and runner (typically `docker` and `nextflow`). -Building an individual component can still be useful, for example when debugging a component for which the build fails or if you want to create a standalone executable for a component to execute it without the need to use `viash`. To build an individual component, `viash build` can be used. Note that the default build directory of this viash base command is `output`, which is not the location where build components will be imported from when integrating them in pipelines. Using the `--output` argument, you can set it to any directory you want, for example: +Building an individual component can be useful when debugging a build failure or creating a standalone executable. Note that the default output directory is `output/`, not `target/` — use `--output` to set the correct location: ```bash -viash build src/filter/do_filter/config.vsh.yaml -o target/native/filter/do_filter/ -p native +viash build src/filter/do_filter/config.vsh.yaml \ + -o target/executable/filter/do_filter/ --engine native ``` ## Containerization One of the key benefits of using Viash is that containers can be created that gather dependencies per component, -which avoids building one container that has to encorporate all dependencies for a pipeline together. The containers for a single component can be reduced in size, defining the minimal requirements to run the component. That being said, building containers from scratch can be labour intensive and error prone, with base containers from reputable publishers often benefiting from improved reliability and security. Hence, a balance has to be made between reducing the container's size and adding many dependencies to a small base container. +which avoids building one container that has to incorporate all dependencies for a pipeline together. The containers for a single component can be reduced in size, defining the minimal requirements to run the component. That being said, building containers from scratch can be labour intensive and error prone, with base containers from reputable publishers often benefiting from improved reliability and security. Hence, a balance has to be made between reducing the container's size and adding many dependencies to a small base container. The preferred containerization setup in OpenPipelines uses the following guidelines: * Choose a base container from a reputable source and use its latest version * Do not use base containers that have not been updated in a while * Use package managers to install dependencies as much as possible -* Avoid building depdencies from source. +* Avoid building dependencies from source. Examples of base containers that are currently being used are: diff --git a/contributing/creating_pipelines.qmd b/contributing/creating_pipelines.qmd index 60e12627..3849fe60 100644 --- a/contributing/creating_pipelines.qmd +++ b/contributing/creating_pipelines.qmd @@ -3,3 +3,337 @@ title: Creating pipelines description: A guide on how to create new workflows order: 30 --- + +OpenPipelines workflows are Nextflow DSL2 pipelines, described to Viash through a `config.vsh.yaml` file. When built, Viash generates the final Nextflow modules from each component's source, and workflows chain those modules together through a shared channel-based state. This page walks through everything needed to create a new workflow from scratch. + +## Workflow structure + +Each workflow lives in its own folder under `src/workflows/[category]/[workflow_name]/`: + +``` +src/workflows/my_category/my_workflow/ +├── config.vsh.yaml # Viash descriptor: arguments, dependencies, test resources +├── main.nf # Nextflow DSL2 workflow logic +├── nextflow.config # Nextflow configuration (resource labels, profile) +└── test.nf # Integration test workflow +``` + +Workflow categories loosely follow the analysis stage, e.g. `rna/`, `integration/`, `annotation/`, `multiomics/`. + +## The config file + +The `config.vsh.yaml` for a workflow is similar to a component config, with a few key differences: it lists `dependencies` (which components and workflows it uses), references a Nextflow script as its resource, and has a `nextflow` runner instead of `docker`/`native`. + +```yaml +name: "my_workflow" +namespace: "workflows/my_category" +description: "Short description of what this workflow does." +authors: + - __merge__: /src/authors/my_name.yaml + roles: [ author ] + +argument_groups: + - name: "Inputs" + arguments: + - name: "--id" + required: true + type: string + description: "Unique identifier for the sample." + example: sample_foo + - name: "--input" + required: true + type: file + description: "Path to the input H5MU file." + example: input.h5mu + + - name: "Outputs" + arguments: + - name: "--output" + required: true + type: file + direction: output + description: "Path to the output H5MU file." + example: output.h5mu + + - name: "Parameters" + arguments: + - name: "--modality" + type: string + default: "rna" + description: "Modality to process." + +dependencies: + - name: filter/filter_with_counts + - name: filter/do_filter + - name: workflows/qc/qc # workflows can depend on other workflows + +resources: + - type: nextflow_script + path: main.nf + entrypoint: run_wf + - type: file + path: /src/workflows/utils/ # shared labels and helper configs + +test_resources: + - type: nextflow_script + path: test.nf + entrypoint: test_wf + - path: /resources_test/pbmc_1k_protein_v3 + +runners: + - type: nextflow +``` + +:::{.callout-tip} +Declare every component and sub-workflow your `main.nf` uses in `dependencies`. Viash uses this list to resolve and include the compiled modules at build time — imports are not written manually in the source `main.nf`. +::: + +## The workflow script + +`main.nf` defines a single Nextflow workflow named `run_wf`. It receives an input channel of `[id, state]` tuples and emits an output channel in the same format. + +### Channel state + +All data flows through a Groovy Map called **state**. Instead of passing individual files between processes, each item in the channel is a tuple `[id, state]` where `state` holds all current values (inputs, outputs, parameters). Components read from the state via `fromState` and write back via `toState`. + +```groovy +workflow run_wf { + take: + input_ch // channel of [id, state] tuples + + main: + output_ch = input_ch + | component_one.run( + fromState: ["input": "input", "modality": "modality"], + toState: ["input": "output"] // component output becomes next input + ) + | component_two.run( + fromState: ["input": "input", "output": "output"], + toState: ["output": "output"] + ) + | setState(["output"]) // trim state to only the declared outputs + + emit: + output_ch +} +``` + +### fromState and toState + +`fromState` maps **state keys → component argument names**. `toState` maps **component output argument names → state keys** for the next step. + +```groovy +| normalize_total.run( + fromState: [ + "input": "input", // state.input → --input + "modality": "modality", // state.modality → --modality + "target_sum": "target_sum" + ], + args: [ + "output_layer": "normalized" // hardcoded args not in state + ], + toState: [ + "input": "output" // component's --output → state.input for next step + ] + ) +``` + +Use a **closure** for `fromState` or `toState` when the mapping involves logic: + +```groovy +| filter_with_counts.run( + fromState: { id, state -> + [ + "input": state.input, + "min_counts": state.min_counts ?: 200, // default if not set + "obs_name_filter": "filter_" + state.modality + ] + }, + toState: { id, output, state -> + state + ["input": output.output] + } + ) +``` + +### Conditional execution + +Use `runIf` to skip a step based on the current state: + +```groovy +| filter_with_scrublet.run( + runIf: { id, state -> !state.skip_doublet_detection }, + fromState: ["input": "input"], + toState: ["input": "output"] + ) +``` + +### Disambiguating repeated components + +If you run the same component more than once in a workflow, give each call a unique `key`: + +```groovy +| filter_with_counts.run( + key: "rna_filter_with_counts", + fromState: [...], + toState: [...] + ) +| filter_with_counts.run( + key: "prot_filter_with_counts", + fromState: [...], + toState: [...] + ) +``` + +### Input validation + +Use a `map` block at the start of the workflow to validate inputs and catch configuration errors early: + +```groovy +workflow run_wf { + take: + input_ch + + main: + output_ch = input_ch + | map { id, state -> + if (!workflow.stubRun) { + assert state.input : "--input is required" + assert state.output : "--output is required" + } + // stash the final output path before intermediate steps overwrite it + [id, state + ["workflow_output": state.output]] + } + | component_one.run( + fromState: ["input": "input"], + toState: ["input": "output"] + ) + | component_two.run( + fromState: ["input": "input", "output": "workflow_output"], + toState: ["output": "output"] + ) + | setState(["output"]) + + emit: + output_ch +} +``` + +:::{.callout-note} +Wrapping assertions in `!workflow.stubRun` avoids failures during stub runs, which are used for workflow graph validation without executing actual processes. +::: + +## The Nextflow config + +`nextflow.config` wires in the shared resource labels and sets the `rootDir` parameter so Viash can resolve relative paths to utilities: + +```groovy +manifest { + nextflowVersion = '!>=20.12.1-edge' +} + +params { + rootDir = java.nio.file.Paths.get("$projectDir/../..").toAbsolutePath().normalize().toString() +} + +includeConfig("${params.rootDir}/src/workflows/utils/labels.config") +``` + +## Resource labels + +Assign CPU and memory requirements by adding labels to a component's `config.vsh.yaml`. The labels are defined in `src/workflows/utils/labels.config` and scale automatically on retry: + +| Label | CPUs | Memory (attempt 1) | +|---|---|---| +| `singlecpu` | 1 | — | +| `lowcpu` | 4 | — | +| `midcpu` | 10 | — | +| `highcpu` | 20 | — | +| `lowmem` | — | 4 GB | +| `midmem` | — | 25 GB | +| `highmem` | — | 50 GB | +| `veryhighmem` | — | 75 GB | + +Labels are declared in the component's runner config, not in the workflow itself — the workflow inherits them automatically from each component it runs. + +## Writing a test + +`test.nf` contains an integration test workflow that runs your workflow end-to-end and validates the output. It follows the same `[id, state]` channel pattern: + +```groovy +include { my_workflow } from "./main.nf" + +workflow test_wf { + output_ch = Channel.fromList([ + [ + "test_sample", + [ + "id": "test_sample", + "input": file("${meta.resources_dir}/pbmc_1k_protein_v3/pbmc_1k_protein_v3_filtered_feature_bc_matrix.h5mu"), + "output": "output.h5mu", + ] + ] + ]) + | my_workflow.run( + fromState: ["id": "id", "input": "input", "output": "output"], + toState: ["output": "output"] + ) + | view { id, state -> + assert state.output.exists() : "Output file does not exist: ${state.output}" + "Output for ${id}: ${state.output}" + } +} +``` + +## Dataflow utility components + +OpenPipelines provides ready-made components for common channel manipulation patterns. These are declared as regular dependencies in your `config.vsh.yaml` and used like any other component in `main.nf`: + +| Component | What it does | +|---|---| +| `dataflow/concatenate_h5mu` | Merge multiple H5MU files into one multi-sample object | +| `dataflow/split_h5mu` | Split one H5MU into per-group files (e.g. by cell type label) | +| `dataflow/merge` | Merge modalities from separate H5MU files into a single object | +| `dataflow/split_modalities` | Split modalities out into separate H5MU files | + +See the [Components reference](../components/) for the full argument list of each. + +## Building and running + +Build all workflows at once: + +```bash +viash ns build --setup cb +``` + +Run a single workflow interactively: + +```bash +nextflow run . \ + -main-script target/nextflow/workflows/my_category/my_workflow/main.nf \ + -profile docker \ + --id my_sample \ + --input path/to/input.h5mu \ + --output path/to/output.h5mu \ + --publish_dir results/ +``` + +Run the workflow's integration test: + +```bash +viash test src/workflows/my_category/my_workflow/config.vsh.yaml +``` + +Run all workflow tests in a namespace: + +```bash +viash ns test --parallel -q workflows/my_category +``` + +:::{.callout-tip} +Always put `--` between the config path and the component/workflow arguments when using `viash run`: + +```bash +viash run src/workflows/my_category/my_workflow/config.vsh.yaml -- \ + --input input.h5mu --output output.h5mu +``` +::: diff --git a/contributing/getting_started.qmd b/contributing/getting_started.qmd index ede740f1..da4b853e 100644 --- a/contributing/getting_started.qmd +++ b/contributing/getting_started.qmd @@ -13,14 +13,14 @@ The OpenPipelines code is hosted on GitHub. To start working on OpenPipelines, y ```bash git clone https://github.com//openpipeline.git cd openpipeline -git remote add upstream https://github.com/openpipeline-bio/openpipeline.git +git remote add upstream https://github.com/openpipelines-bio/openpipeline.git ``` # SSH ```bash git clone git@github.com:/openpipeline.git cd openpipeline -git remote add upstream https://github.com/openpipeline-bio/openpipeline.git +git remote add upstream https://github.com/openpipelines-bio/openpipeline.git ``` ::: diff --git a/contributing/project_structure.qmd b/contributing/project_structure.qmd index ec869fd7..fb1e3a1b 100644 --- a/contributing/project_structure.qmd +++ b/contributing/project_structure.qmd @@ -17,7 +17,7 @@ When cloning a fresh repository, there will be no `target` folder present. This ## Versioning and branching strategy {#sec-versioning} -OpenPipeline tries to use of [semantic versioning](https://semver.org/) to govern changes between versions. An release of openpipelines uses a version number in the format `MAJOR.MINOR.PATCH`. Currenly, openpipelines is still at major version `0.x.y`, meaning that public-facing breaking changes are possible on `MINOR` releases. These breaking changes will be documented in a dedicated section of the CHANGELOG that is published with each release. A `PATCH` release (i.e. a release where the `MAJOR` and `MINOR` version number stay the same), is used to resolve bugs with the pipeline but should not introduce breaking changes. Keep in mind that patches might introduce behavioral changes that may look breaking but are actually rectifying changes that were inadvertently introduced previously (and were in fact also 'breaking changes'). In this case, a bug can also be released without changing the `MINOR` version, in a `PATH` release. +OpenPipeline tries to use of [semantic versioning](https://semver.org/) to govern changes between versions. An release of openpipelines uses a version number in the format `MAJOR.MINOR.PATCH`. OpenPipelines follows semantic versioning. At major version 4.x, public-facing breaking changes are documented in the CHANGELOG on each `MINOR` release. These breaking changes will be documented in a dedicated section of the CHANGELOG that is published with each release. A `PATCH` release (i.e. a release where the `MAJOR` and `MINOR` version number stay the same), is used to resolve bugs with the pipeline but should not introduce breaking changes. Keep in mind that patches might introduce behavioral changes that may look breaking but are actually rectifying changes that were inadvertently introduced previously (and were in fact also 'breaking changes'). In this case, a bug can also be released without changing the `MINOR` version, in a `PATH` release. Between releases, development progress is tracked on Git branches. A git branch represents a snapshot of a codebase in time, to which changes can be added (i.e. committed). Eventually, all new feature or bugfixes must be reconsiled into a single branch so that a new release can be created. This process is called merging and the process of requesting the merging of two branches is called a pull request. Openpipelines follows the convention that the target branch for all pull requests is the `main` branch. Thus, the `main` branch contains the latest changes for the code and it can be considered the development branch. diff --git a/contributing/running_tests.qmd b/contributing/running_tests.qmd index 73969495..eaac9c88 100644 --- a/contributing/running_tests.qmd +++ b/contributing/running_tests.qmd @@ -4,34 +4,147 @@ description: How to run component and integration tests. order: 30 --- +## Test structure -## Fetch the test data -The input data that is needed to run the tests will need to be downloaded from the openpipelines Amazon AWS s3 bucket first. -To do so, the `download/sync_test_resource` component can be used, which will download the data to the correct location (`resources_test`) by default. +Every component has its tests defined alongside its source files: + +``` +src/my_namespace/my_component/ +├── config.vsh.yaml +├── script.py / script.sh / script.R +├── test.sh # component test script +└── test_data/ + └── test_data_script.sh # script to generate local test inputs +``` + +Workflows use a Nextflow test workflow instead: + +``` +src/workflows/my_category/my_workflow/ +├── config.vsh.yaml +├── main.nf +└── test.nf # integration test workflow +``` + +`viash test` automatically picks up `test.sh` (for components) or `test.nf` (for workflows) — no extra configuration needed. + +## Generating test data locally + +Rather than downloading test data from S3, the preferred approach for new components is to generate test inputs using a shell script. This makes your tests self-contained and reproducible without network access. + +Create `test_data/test_data_script.sh` alongside your component: + +```bash +#!/bin/bash +set -eo pipefail + +# Run from the component directory +cd "$(dirname "$0")" + +# Generate a minimal H5MU test file using an existing converter or Python +python3 - <<'EOF' +import mudata as mu +import anndata as ad +import numpy as np +import scipy.sparse + +n_obs, n_vars = 50, 100 +X = scipy.sparse.random(n_obs, n_vars, density=0.3, format="csr") +adata = ad.AnnData(X=X) +adata.obs_names = [f"cell_{i}" for i in range(n_obs)] +adata.var_names = [f"gene_{i}" for i in range(n_vars)] +mdata = mu.MuData({"rna": adata}) +mdata.write_h5mu("test_input.h5mu") +print("Written test_input.h5mu") +EOF +``` + +Run it once before testing: + +```bash +bash src/my_namespace/my_component/test_data/test_data_script.sh +``` + +## Writing a component test + +`test.sh` runs the component and asserts the output is correct. A minimal pattern: + +```bash +#!/bin/bash +set -eo pipefail + +# viash injects $meta_resources_dir and $meta_temp_dir at test time +cd "$meta_temp_dir" + +echo ">>> Running component" +viash run "$meta_resources_dir/config.vsh.yaml" -- \ + --input "$meta_resources_dir/test_data/test_input.h5mu" \ + --output output.h5mu + +echo ">>> Checking output exists" +[[ -f output.h5mu ]] || { echo "Output file missing"; exit 1; } + +echo ">>> Checking output is valid H5MU" +python3 -c " +import mudata as mu +mdata = mu.read_h5mu('output.h5mu') +assert 'rna' in mdata.mod, 'rna modality missing' +print('obs:', mdata.n_obs, 'vars:', mdata.mod['rna'].n_vars) +" + +echo ">>> All tests passed" +``` + +## Fetch test data from S3 + +For CI and components that use shared test resources, download the data from the OpenPipelines S3 bucket: ```bash viash run src/download/sync_test_resources/config.vsh.yaml ``` +This downloads to `resources_test/` by default, which is where test configs reference shared fixtures. ## Run component tests -To build and run tests for individual component that you are working on, use [viash test](https://viash.io/api/commands/test/) with the `config.vsh.yaml` of the component you would like to test. -For example: + +Test a single component (builds on the fly, no separate build step needed): + +```bash +viash test src/my_namespace/my_component/config.vsh.yaml +``` + +By default this uses the first declared engine, which is Docker for most components. To test without Docker: + +```bash +viash test src/my_namespace/my_component/config.vsh.yaml --engine native +``` + +Test all components in a namespace in parallel: ```bash -viash test src/convert/from_10xh5_to_h5mu/config.vsh.yaml +viash ns test --parallel -q my_namespace ``` -Keep in mind that when no platform is passed to `viash test`, it will use the first platform that is specified in the config, which is `docker` for most of the components in openpipelines. Use `-p native` for example if you do not want to use docker. -It is also possible to execute the tests for all components in each namespace using: +## Run workflow integration tests + +Test a single workflow end-to-end using its `test.nf`: + +```bash +viash test src/workflows/my_category/my_workflow/config.vsh.yaml +``` + +For workflows that provide a separate integration test script alongside `main.nf`: ```bash -viash ns test --parallel -q convert +bash src/workflows/ingestion/cellranger_mapping/integration_test.sh ``` -## Run integration tests -Individual integration tests can be run by using the `integration_test.sh` scripts for a pipeline, located next to the `main.nf` in the `src/workflows` folder. +## Run the full test suite ```bash -src/workflows/ingestion/cellranger_demux/integration_test.sh +viash ns test --parallel ``` + +:::{.callout-note} +The full suite requires all test data to be present in `resources_test/`. Run `sync_test_resources` first if you have not already done so. +::: diff --git a/fundamentals/architecture.qmd b/fundamentals/architecture.qmd index 997f96b3..1ab753d7 100644 --- a/fundamentals/architecture.qmd +++ b/fundamentals/architecture.qmd @@ -1,987 +1,152 @@ --- title: Architecture -description: Structure of the project +description: How OpenPipelines fits into a single-cell project order: 30 -d2: - layout: elk --- -OpenPipeline is a pipeline for the processing of multimodal single-cell data that scales to a great many of samples. Covering the architecture requires us to explain many angles, including: what the expected inputs and outputs are for each workflow are, how do the workflows relate to each other, and what the state of the data is at each step of the pipeline. Here is an overview of the general steps involved in processing sequencing data into a single integrated object. We will discuss each of the steps further below. +OpenPipelines provides modular, best-practice Nextflow workflows for single-cell data processing — from raw sequencing output to a batch-corrected, clustered dataset. The diagram below shows the five pipeline stages a single-cell dataset flows through, from raw experimental data to integrated and annotated results, together with the cross-cutting QC-reporting and spatial branches. ```{mermaid} %%| label: fig-architecture -%%| fig-cap: Overview of the steps included in OpenPipeline for the analysis of single cell multiomics data. -flowchart TD - ingest["Ingestion"] --> split --> unimodalsinglesample["Unimodal Single Sample Processing"] --> concat --> unimodalmultisample["Unimodal Multi Sample Processing"] --> merging --> integation_setup["Integration Setup"] --> integration["Integration"] --> downstreamprocessing["Downstream Processing"] +%%| fig-width: 7 +%%| fig-cap: "Overview of the OpenPipelines architecture. A single-cell dataset flows through five pipeline stages — demultiplexing & mapping, ingestion & conversion, QC & preprocessing, integration & clustering, and downstream analysis — starting from raw experimental data (grey). Data can enter either as raw FASTQ (after demultiplexing) or as pre-built count matrices. Cross-cutting QC reporting and the spatial branch feed off the core flow." +%%{init: {"theme":"base", "themeVariables":{"fontSize":"14px"}}}%% +flowchart TD + classDef external fill:#f5f5f5,stroke:#aaaaaa,color:#444 + classDef map fill:#e1d5e7,stroke:#9673a6,color:#000 + classDef ingest fill:#d5e8d4,stroke:#82b366,color:#000 + classDef process fill:#dae8fc,stroke:#6c8ebf,color:#000 + classDef integ fill:#ffe6cc,stroke:#d6b656,color:#000 + classDef down fill:#f8cecc,stroke:#b85450,color:#000 + classDef spatial fill:#d0f0f5,stroke:#3a9db5,color:#000 + classDef qc fill:#fff3cd,stroke:#c9a227,color:#000 + + wetlab["Experiment & sequencing
10x Chromium
BD Rhapsody
Xenium (spatial)"]:::external + fastq["FASTQ files
or pre-built count matrices"]:::external + + map["1. Demultiplexing & mapping
demux (BCL to FASTQ)
cellranger_multi
bd_rhapsody
star_align"]:::map + ingest["2. Ingestion & conversion
from_cellranger_multi_to_h5mu
from_xenium_to_h5mu
to MuData (H5MU) per sample"]:::ingest + process["3. QC & preprocessing
filter_with_counts
cellbender_remove_background
clr / normalize / HVF
pca"]:::process + integ["4. Integration & clustering
harmony_leiden
scvi_leiden
totalvi_leiden
bbknn
to Leiden clusters + UMAP"]:::integ + down["5. Downstream analysis
scanvi_scarches
celltypist
singler
lianapy
deseq2"]:::down + qcreport["QC reporting
generate_qc_report"]:::qc + spatial["Spatial
from_xenium_to_h5mu
spatial_neighborhood_graph
spatial_autocorr"]:::spatial + + wetlab --> map --> ingest --> process --> integ --> down + fastq -.-> ingest + process -.-> qcreport + ingest -.-> spatial + spatial -.-> down ``` -```{mermaid} -flowchart TB - subgraph ingestion - direction TB - subgraph cellranger_multi - direction TB - mapping_10x --> convert_to_h5mu_10x - mapping_10x[mapping] - convert_to_h5mu_10x[convert_to_h5mu] - end - subgraph bdrhap_v1 - direction TB - mapping_bd1 --> convert_to_h5mu_bd1 - mapping_bd1[mapping] - convert_to_h5mu_bd1[convert_to_h5mu] - end - subgraph bdrhap_v2 - direction TB - mapping_bd2 --> convert_to_h5mu_bd2 - mapping_bd2[mapping] - convert_to_h5mu_bd2[convert_to_h5mu] - end - cellranger_multi:::subwf - bdrhap_v1:::subwf - bdrhap_v2:::subwf - end - ingestion:::wf - subgraph process_samples - split_modalities --> rna_singlesample & prot_singlesample & gdo_singlesample & atac_singlesample & other_modalities --> concat --> process_batches - split_modalities - rna_singlesample - prot_singlesample - gdo_singlesample - atac_singlesample - other_modalities - concat - subgraph process_batches - direction LR - split_modalities2 --> rna_multisample & prot_multisample & atac_multisample & other_modalities2 --> merge - split_modalities2[split_modalities] - other_modalities2[other_modalities] - rna_multisample - prot_multisample - merge - end - process_batches:::subwf - atac_multisample - end - process_samples:::wf - raw_counts --- ingestion --> raw_h5mu - raw_h5mu --- process_samples --> processed_h5mu - subgraph integration - direction LR - integration_method --> find_neighbors --> leiden --> umap - integration_method -.- intmeth - integration_method - find_neighbors - leiden - umap - subgraph intmeth [integration_method] - bbknn - harmony - scanorama - scvi - totalvi - scgpt_integration - end - intmeth:::info - end - integration:::wf - processed_h5mu --- integration ---> integrated_h5mu - subgraph celltype_annotation - direction TB - integration_method2[integration_method] - celltypist - scanvi - scgpt_annotation - onclass - svm - randomforest - pynndescent_knn - consensus_voting - integration_method2 --> pynndescent_knn --> consensus_voting - celltypist & scanvi & scgpt_annotation & onclass & svm & randomforest --> consensus_voting - end - reference_atlas --> celltype_annotation - celltype_annotation:::wf - integrated_h5mu --- celltype_annotation --> annotated_h5mu - - classDef wf fill:#f0f0f0,stroke:#525252 - classDef subwf fill:#d9d9d9,stroke:#525252 - classDef info fill:#f0f0f0,stroke:#525252,stroke-dasharray: 4 4 -``` - -1. [Ingestion](#ingestion): Convert raw sequencing data or count tables into MuData data for further processing. -2. [Splitting modalities](#sec-splitting): Creating several MuData objects, one per modality, out of a multimodal input sample. -3. [Unimodal Single Sample Processing](#sec-single-sample): tools applied to each modality of samples individually. Mostly involves the selection of true from false cells. -4. [Unimodal Multi Sample Processing](#sec-multisample-processing): steps that require information from all samples together. Processing is still performed per-modality. -5. [Merging](#sec-merging): Creating one MuData object from several unimodal MuData input files. -6. [Initializing Integration](#sec-initializing-integration): Performs dimensionality reduction and cell type clustering on non-integrated samples. These are popular steps that would otherwise be executed manually or they provide input for downstream integration methods. -7. [Integration](#sec-intergration): The alignment of cell types across samples. Can be performed per modality or based on multiple modalities. -8. Downstream Processing: Extra analyses performed on the integrated dataset and conversion to other file formats. - -# Available workflows -The structure of the sections that have been laid out below follow a logical grouping of the processing according to the state _of the data_. However, even though this grouping makes sense from a data perspective, it does not mean that a workflow exist for each section. For example, the [processing a single sample](#sec-single-sample) section describes how processing of single sample is performed as part of the [full pipeline](#sec-full-pipeline), but there is no `singlesample` workflow that a user can execute. The inverse is also possible: while there exists an [multisample](../components/workflows/multiomics/mutlisample.qmd) pipeline, it's functionality is not limited to what has been described in the section [Multisample Processing](#sec-multisample-processing). This section lists all the available workflows and will try to describe the link with the relevant sections below. - -## Ingestion workflows -All of the following workflows from the ingestion namespace have been discussed in more detail in the [ingestion](#sec-ingestion) section: - -* [ingestion/bd_rhapsody](../components/workflows/ingestion/bd_rhapsody.qmd) -* [ingestion/cellranger_mapping](../components/workflows/ingestion/cellranger_mapping.qmd) -* [ingestion/cellranger_multi](../components/workflows/ingestion/cellranger_multi.qmd) -* [ingestion/demux](../components/workflows/ingestion/demux.qmd) -* [ingestion/make_reference](../components/workflows/ingestion/make_reference.qmd) - - -## Multiomics workflows -There exists no `singlesample` workflow. However, the `prot_singlesample` and `rna_singlesample` pipelines do exist and they map identically to the functionality described in the [single-sample antibody capture processing](#sec-single-sample-adt) and [single-sample gene expression processing](#sec-single-sample-gex) sections respectively. If you would like to process your samples as described in the [unimodal single sample processing](#sec-single-sample) section, you can execute both workflows in tandem for the two modalities. - -Contrary to the workflows for single sample processing, there exists a [multiomics/multisample](../components/workflows/multiomics/multisample.qmd) workflow. However this workflow is not just the [multiomics/prot_multisample](../components/workflows/multiomics/prot_multisample.qmd) and [multiomics/rna_multisample](../components/workflows/multiomics/rna_multisample.qmd) workflows that have been combined. Instead, it combines the [multiomics/prot_multisample](../components/workflows/multiomics/prot_multisample.qmd), [multiomics/rna_multisample](../components/workflows/multiomics/rna_multisample.qmd) and [multiomics/integration/initialize_integration](../components/workflows/multiomics/integration/initialize_integration.qmd) workflows. The purpose of this pipeline is to provide an extra 'entrypoint' into the full pipeline that skips the singlesample processing, allowing reprocessing samples that have already been processed before. A popular usecase is to manually select one or more celltypes which need to be processed again or the integration of observations from multiple experiments into a single dataset. Keep in mind that concatenation is not included in the multisample pipeline, so when multiple input files are specified they are processed in parallel. If you would like to integrate multiple experiments, you need to first concatenate them in a seperate step: - -```{.d2} -file_input_1: "Experiment 1\n(multisample)" { - shape: page -} - -file_input_2: "Experiment 2\n(multisample)" { - shape: page -} - -file_output: "Output" { - shape: page -} - -pipeline_out_1: "Full pipeline \n integration \n..." { - shape: parallelogram - style.stroke-dash: 5 -} - -pipeline_out_2: "Full pipeline \n integration \n..." { - shape: parallelogram - style.stroke-dash: 5 -} - -concat: "Concatenation" { - shape: parallelogram -} - -multisample: "Multisample" { - shape: parallelogram -} - -pipeline_out_1 -> file_input_1 -pipeline_out_2 -> file_input_2 -file_input_1 -> concat <- file_input_2 - -concat -> multisample -> file_output - -style: { - fill: "#FCFCFC" -} -``` - -## The "full" pipeline -The name of this pipeline is a bit of a misnomer, because it does not include all the steps from ingestion to integration. As will be discussed in the [ingestion](#sec-ingestion) section, which ingestion strategy you need is dependant on your technology provider and the chosen platform. For [integration](#sec-integration-methods), there exist many methods and combination of methods, and you may wish to choose which integration methods are applicable for your usecase. As a consequence, these two stages in the analysis of single-cell need to be executed seperatly and not as part of a single unified pipeline. All other steps outlined below on the other hand are included into the "full" pipeline, which can therefore be summarized in the following figure: - -```{mermaid} -%%| label: fig-full-pipeline -%%| fig-cap: Overview of the steps included in the full pipelines from OpenPipeline. -flowchart TD - split --> unimodalsinglesample["Unimodal Single Sample Processing"] --> concat --> unimodalmultisample["Unimodal Multi Sample Processing"] --> merging --> integation_setup -``` - -## Integration workflows -For each of the integration methods (and their optional combination with other tools), a seperate pipeline is defined. More information for each of the pipelines is available in the [integration methods section](#sec-integration-methods). - -* [multiomics/integration/bbknn_leiden](../components/workflows/multiomics/integration/bbknn_leiden.qmd) -* [multiomics/integration/harmony_leiden](../components/workflows/multiomics/integration/harmony_leiden.qmd) -* [multiomics/integration/scanorama_leiden](../components/workflows/multiomics/integration/scanorama_leiden.qmd) -* [multiomics/integration/scvi_leiden](../components/workflows/multiomics/integration/scvi_leiden.qmd) -* [multiomics/integration/totalvi_leiden](../components/workflows/multiomics/integration/totalvi_leiden.qmd) -* [multiomics/integration/initialize_integration](../components/workflows/multiomics/integration/initialize_integration.qmd) - - -# Important dataflow components -While most components included in openpipelines are involved in data analysis, the sole purpose of other components is to facilitate data flow throughout the pipelines. In a workflow, output for a component is written to disk after it is done performing its task, and is read back in by the next component. However, the relation between the component and the next component is not always a clear one to one relationship. For example: some tools are capable of analyzing a single sample, while others require the input of all samples together. Additionally, not only are the input requirement of tools limiting, performance also needs to be taken into account. Tasks which are performed on each sample separately can be executed in parallel, while if the same task is performed on a single file that contains the data for all samples. In order to facilitate one-to-many or many-to-one relations between components and to allow parallel execution of tasks, component that are specialized in dataflow were implemented. - -## Splitting modalities {#sec-splitting} -We refer to splitting modalities when multimodal MuData file is split into several unimodal MuData files. The number of output files is equal to the number of modalities present in the input file. Splitting the modalities works on MuData files containing data for multiple samples or for single-sample files. - -~~~{.d2} -file_input: MuData Input{ - shape: page -} - -file_input.mudata_input: |||md - ``` - └─.mod - └─ rna - └─ prot - └─ ... - ``` -||| -file_input.style.font-size: 20 - -file_output_rna: MuData Output\nGene Expression{ - shape: page -} -file_output_rna.mudata_output_rna: |||md - ``` - └─.mod - └─ rna - ``` -||| -file_output_rna.style.font-size: 20 - -file_output_prot: MuData Output\nAntibody Capture{ - shape: page -} -file_output_prot.mudata_output_prot: |||md - ``` - └─.mod - └─ prot - ``` -||| -file_output_prot.style.font-size: 20 - -file_output_other: MuData Output\nOther{ - shape: page -} -file_output_other.mudata_output_other: |||md - ``` - └─.mod - └─ ... - ``` -||| -file_output_other.style.font-size: 20 - -split_modalities: Split Modalities { - shape: parallelogram -} - - -file_input -> split_modalities -split_modalities -> file_output_rna -split_modalities -> file_output_prot -split_modalities -> file_output_other - -style: { - fill: "#FCFCFC" -} - -~~~ - -## Merging of modalities {#sec-merging} -Merging refers to combining multiple files with data for one modality into a single output file that contains all input modalities. It is the inverse operation of splitting the modalities. - -~~~{.d2} -file_output: MuData Output{ - shape: page -} - -file_output.mudata_input: |||md - ``` - └─.mod - └─ rna - └─ prot - └─ ... - ``` -||| -file_output.style.font-size: 20 - -file_input_rna: MuData Input\nGene Expression{ - shape: page -} -file_input_rna.mudata_input_rna: |||md - ``` - └─.mod - └─ rna - ``` -||| -file_input_rna.style.font-size: 20 - -file_input_prot: MuData Input\nAntibody Capture{ - shape: page -} -file_input_prot.mudata_input_prot: |||md - ``` - └─.mod - └─ prot - ``` -||| -file_input_prot.style.font-size: 20 - -file_input_other: "MuData Input\nOther"{ - shape: page -} -file_input_other.mudata_input_other: |||md - ``` - └─.mod - └─ ... - ``` -||| -file_input_other.style.font-size: 20 - -merge_modalities: Merge Modalities { - shape: parallelogram -} - -file_input_rna -> merge_modalities -file_input_prot -> merge_modalities -file_input_other -> merge_modalities -merge_modalities -> file_output - -style: { - fill: "#FCFCFC" -} - -~~~ - -## Concatenation of samples - -Joining of observations for different samples, stored in their respective MuData file, into a single MuData file for all samples together is called sample concatenation. In practice, this operation is performed for each modality separately. An extra column (with default name `sample_id`) is added to the annotation of the observations (`.obs`) to indicate where each observation originated from. - -~~~{.d2} -file_output: MuData Output{ - shape: page -} - -file_output.mudata_input: |||md - ``` - └─.mod - └─ rna - └─ prot - └─ vdj - - └─.obs - [sample_id] - ``` -||| -file_output.style.font-size: 20 - -file_input_sample1: "MuData Input\nSample 1"{ - shape: page -} -file_input_sample1.mudata_input_sample1: |||md - ``` - └─.mod - └─ rna - └─ vdj - ``` -||| -file_input_sample1.style.font-size: 20 - -file_input_sample2: "MuData Input\nSample 2"{ - shape: page -} -file_input_sample2.mudata_input_prot: |||md - ``` - └─.mod - └─ rna - └─ prot - ``` -||| -file_input_sample2.style.font-size: 20 - -file_input_sample3: "MuData Input\nSample 3"{ - shape: page -} -file_input_sample3.mudata_input_sample3: |||md - ``` - └─.mod - └─ rna - ``` -||| -file_input_sample3.style.font-size: 20 - -concatenate_samples: Concatenation { - shape: parallelogram -} - -file_input_sample1 -> concatenate_samples -file_input_sample2 -> concatenate_samples -file_input_sample3 -> concatenate_samples -concatenate_samples -> file_output - -style: { - fill: "#FCFCFC" -} -~~~ - -Special care must be taken when considering annotations for observations and features while concatenating the samples. Indeed, the data from different samples can contain conflicting information. Openpipeline's `concat` component provides an argument `other_axis_mode` that allows a user to specify what happens when conflicting information is found. The `move` option for this argument is the default behavior. In this mode, each annotation column (from `.obs` and `.var`) is compared across samples. When no conflicts are found or the column is unique for a sample, the column is added output object. When a conflict does occur, all of the columns are gathered from the samples and stored into a dataframe. This dataframe is then stored into `.obsm` for annotations for the observations and `.varm` for feature annotations. This way, a user can have a look at the conflicts and decide what to do with them. - -# Ingestion {#sec-ingestion} - -Ingestion is the conversion of raw sequencing data or count tables into MuData objects that can be used for further processing. - -```{mermaid} -flowchart LR - RawCounts1["Raw counts"] - BCL[/"BCL"/] - Demux[/"Demultiplexing"/] - Fastq["Fastq"] - Ref["Reference"] - Mapping[/"Mapping"/] - RawDir["Raw out"] - Convert[/"Convert"/] - RawCounts1["Raw counts"] - BCL --> Demux --> Fastq - Fastq & Ref --> Mapping --> RawDir --> Convert --> RawCounts1 -``` - -Demultiplexing refers to a two-step process: - -(@) The conversion of the binary base call (BCL) files, output by the sequencing machines, into the text-based FASTQ format. -(@) The sorting of reads into different FASTQ files for different libraries pooled together into a single sequencing run. - -In order to perform demultiplexing, several tools have been made available in the [demux](../components/workflows/ingestion/demux.qmd) workflow, where the `--demultiplexer` can be used to choose your demultiplexer of choice. Currently, three options have been made available: - -* [bcl2fastq(2)](../components/modules/demux/bcl2fastq.qmd): a legacy tool from Illumina that has been replaced by BCL Convert -* [BCL Convert](../components/modules/demux/bcl_convert.qmd): general demultiplexing software by Illumina. -* Cellranger's [mkfastq](../components/modules/demux/cellranger_mkfastq.qmd): a wrapper around BCL Convert that provides extra convenience features for the processing of 10X single-cell data. - -The alignment of reads from the FASTQ files to an appropriate genome reference is called mapping. The result of the mapping process are tables that count the number of times a read has been mapped to a certain feature and metadata information for the cells (observations) and features. There are different format that can be used to store this information together. Since OpenPipeline uses [MuData](./concepts.qmd#sec-common-file-format) as a common file format throughout its pipelines, a conversion to MuData is included in the mapping pipelines. The choice between workflows for mapping is dependant on your single-cell library provider and technology: - -* For DB Genomics libraries, the [BD Rhapsody](../components/workflows/ingestion/bd_rhapsody.qmd) pipeline can be used. -* For 10X based libraries, either [cellranger count](../components/workflows/ingestion/cellranger_mapping.qmd) or [cellranger multi](../components/workflows/ingestion/cellranger_multi.qmd) is provided. For more information about the differences between the two and when to use which mapping software, please consult the [10X genomics website](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/using/multi#when-to-use-multi). - -## Creating a transcriptomics reference -Mapping reads from the FASTQ files to features requires a reference that needs to be provided to the mapping component. Depending on the usecase, you might even need to provide references specific for the modalities that you are trying to analyze. For gene expression data, the reference is a reference genome, together with its appropriate gene annotation. A genome reference is often indexed in order to improve the mapping speed. Additionally, some mapping frameworks provided by the single-cell technology providers require extra preprocessing of the reference before they can be used with their worklow. OpenPipelines provides a [make_reference](../components/workflows/ingestion/make_reference.qmd) that allows you to create references in many formats which can be used to map your reads to. - - -# Processing a single sample {#sec-single-sample} -Some processing can (or must) be performed without concatenating samples together. Even when having the choice, adding a tool to the single-sample processing is preferred because multiple samples can be processed in parallel, improving the processing speed. In general, the processing is modality specific, meaning that a multi-modal sample is first split into its unimodal counterparts. As described in the [multi-sample processing](#sec-multisample-processing), the resulting files are _not_ merged back together after the single-sample processing is done. Instead, the output files for all samples are gathered per modality and concatenated to create a multi-sample unimodal object. - - -~~~{.d2} -file_input: Input File{ - shape: page -} - -split: Split Modalities { - shape: parallelogram -} - - -unimodal_gex: "Unimodal Processing\nGene Expression"{ - shape: parallelogram -} - -unimodal_prot: "Unimodal Processing\nAntibody Capture"{ - shape: parallelogram -} - - -multisample: "To Multi-sample Processing"{ - shape: parallelogram - style.stroke-dash: 5 -} - - -file_input -> split -split -> unimodal_gex -split -> unimodal_prot -split -> multisample - -unimodal_gex -> multisample { - style.stroke-dash: 3 -} - -unimodal_prot -> multisample { - style.stroke-dash: 3 -} - -style: { - fill: "#FCFCFC" -} - -~~~ - -## Single-sample Gene Expression Processing {#sec-single-sample-gex} -Single-sample gene expression processing involves two steps: removing cells based on count statistics and flagging observations originating from doublets. - -The removal of cells based on basic count statistics is split up into two parts: first, cells are flagged for removal by [filter_with_counts](../components/modules/filter/filter_with_counts.qmd). It flags observations based on several thresholds: - -* The number of genes that have a least a single count. Both a maximum and minimum number of genes for a cell to be removed can be specified. -* The percentage of read counts that originated from a mitochodrial genes. Cells can be filtered based on both a maximum or minimum fraction of mitochondrial genes. -* The minimum or maximum total number of counts captured per cell. Cells with 0 total counts are always removed. - -Flagging cells for removal involved adding a boolean column to the `.obs` dataframe. After the cells have been flagged for removal, the cells are actually filtered using [do_filter](../components/modules/filter/do_filter.qmd), which reads the values in `.obs` and removed the cells labeled `True`. This applies the general phylosophy of "separation of concerns": one component is responsible for labeling the cells, another for removing them. This keeps the codebase for a single component small and its functionality testable. - -The next and final step in the single-sample gene expression processing is doublet detection using [filter_with_scrublet](../components/modules/filter/filter_with_scrublet.qmd). Like `filter_with_counts`, it will not remove cells but add a column to `.obs` (which have the name `filter_with_scrublet` by default). The single-sample GEX workflow will not remove not be removed during the processing (hence no `do_filter`). However, you can choose to remove them yourself before doing your analyses by applying a filter with the column in `.obs` yourself. - -~~~{.d2} -direction: right -file_input: "Input File" { - shape: page -} - -count_filtering: "Count Filtering" { - shape: parallelogram -} - -doublet_removal: "Doublet Removal" { - shape: parallelogram -} - -doublet_removal.filter_with_scrublet { - shape: parallelogram -} - -file_output: "Output File" { - shape: page -} - -count_filtering.filter_with_counts { - shape: parallelogram -} - -count_filtering.do_filter { - shape: parallelogram -} - -file_input -> count_filtering -count_filtering -> doublet_removal -doublet_removal -> file_output - - -style: { - fill: "#FCFCFC" -} -~~~ - -## Single-sample Antibody Capture Processing {#sec-single-sample-adt} -The process of filtering antibody capture data is similar to the filtering in [the single-sample gene-expression processing](#sec-single-sample-gex), but without doublet detection. In some particular cases you can use your ADT data to perform doublet detection using for example cell-type maskers. More information can be found in [the single-cell best practices book](https://www.sc-best-practices.org/surface_protein/doublet_detection.html). +## Demultiplexing -~~~{.d2} -direction: right -file_input: "Input File" { - shape: page -} +If your sequencing provider delivers raw Illumina BCL files, convert them to per-sample FASTQ files at the start of the pipeline. This step is **often already performed upstream** by your sequencing provider, but is also available within OpenPipelines. -count_filtering: "Count Filtering" { - shape: parallelogram -} +The [`ingestion/demux`](../components/workflows/ingestion/demux.qmd) workflow supports three demultiplexers selected via `--demultiplexer`: -file_output: "Output File" { - shape: page -} +- **bcl2fastq** — legacy Illumina tool +- **BCL Convert** — current Illumina general-purpose demultiplexer +- **cellranger mkfastq** — 10x Genomics wrapper around BCL Convert with convenience features for 10x libraries -count_filtering.filter_with_counts { - shape: parallelogram -} +If you already have FASTQ files or a pre-built count matrix, skip directly to [Ingestion](#ingestion). -count_filtering.do_filter { - shape: parallelogram -} +## Ingestion -file_input -> count_filtering -count_filtering -> file_output +Every cell in a single-cell experiment is tagged with a unique barcode before sequencing. Ingestion aligns the resulting short reads back to a reference genome, uses the barcode to assign each read to a cell, and uses the UMI (Unique Molecular Identifier) to count each RNA molecule exactly once — removing the amplification bias from PCR. The result is a **count matrix**: cells × genes, with each entry being the number of RNA molecules detected. +OpenPipelines stores this count matrix — along with cell and gene metadata and any additional modalities (ADT, VDJ, ATAC) — in a single **H5MU file per sample**. This per-sample H5MU is the unit that flows through all subsequent steps. The right ingestion workflow depends on your sequencing platform and library type. See [Ingestion](../user_guide/ingestion.qmd) for workflow selection, reference genome setup, and examples. -style: { - fill: "#FCFCFC" -} +## Process samples -~~~ +Raw count matrices contain not only real cells but also empty droplets (barcodes that captured ambient RNA), damaged cells (high mitochondrial fraction), and doublets (two cells sequenced as one). The `process_samples` workflow handles all per-sample and multi-sample processing in a single command. -# Multisample Processing {#sec-multisample-processing} -After the processing of individual samples has been concluded, samples can be concatenated for further processing. Like with the single-sample processing the multisample processing is not performed on multimodal objects, but each modality separately in order to tailor for the specific modality in question. This means that the result from the singlesample processing is merged together per-modality to create unimodal multisample objects. After processing each modality, all of the modalities can finally be merged and a single object is created that is ready for the integration. +### Per-sample QC (single-sample processing) -~~~{.d2} -direction: down +Each sample is first split by modality. Per modality, cells are flagged for removal based on: +- **Total counts** — barcodes with zero counts or counts outside a defined range are removed. +- **Number of detected genes** — minimum and maximum gene count thresholds filter empty droplets and low-quality cells. +- **Mitochondrial fraction** — cells with a high fraction of reads mapping to mitochondrial genes are filtered out (damaged cells lose nuclear RNA while mitochondrial RNA leaks into the cytoplasm). -input_gex_2: "Processed Sample 2\nGene Expression" { - shape: page -} -input_adt_2: "Processed Sample 1\nAntibody Capture" { - shape: page -} +For gene expression (RNA) data, **doublet detection** is then run using Scrublet, which scores each cell for its likelihood of being a doublet. Doublet scores are added to `.obs` rather than immediately removed, letting users apply their own threshold. -input_gex_1: "Processed Sample 1\nGene Expression" { - shape: page -} -input_adt_1: "Processed Sample 2\nAntibody Capture" { - shape: page -} +For antibody capture (ADT/CITE-seq) data, the same count-based filtering is applied without doublet detection. -input_other_1: "Processed Sample 1\nOther Modality" { - shape: page -} -input_other_2: "Processed Sample 2\nOther Modality" { - shape: page -} +### Multi-sample normalization and dimensionality reduction +After per-sample QC, samples are **concatenated together** — not normalized per sample — because normalization methods need to see variation across the full cohort. A `sample_id` column is added to `.obs` to record the origin of each cell. When conflicting metadata is found across samples, the conflicting columns are moved to `.obsm` (observations) or `.varm` (variables) for inspection. -concat_gex: "Concatenate\nGene Expression" { - shape: parallelogram -} +The concatenated object then goes through: -concat_adt: "Concatenate\nAntibody Capture" { - shape: parallelogram -} +1. **Normalization** — each cell is scaled to a fixed total count (e.g. 10 000), then log1p-transformed (`X = ln(X + 1)`), stabilizing variance and making relative changes interpretable. +2. **Highly variable feature (HVF) selection** — typically 2 000–3 000 genes with the most variation across cells are retained; the rest are de-prioritized. Driven by the Seurat method by default. +3. **PCA** — Principal Component Analysis compresses the high-dimensional gene expression space into a smaller set of components that capture the major axes of variation. All integration methods operate in this reduced space. -concat_other: "Concatenate\nOther Modality" { - shape: parallelogram -} +For large cohorts, [`process_batches`](../components/workflows/multiomics/process_batches.qmd) adds a batching layer that processes groups of samples through multi-sample normalization in parallel before merging, reducing peak memory. See [Processing](../user_guide/processing.qmd) for full details and examples. -multisample_gex: "Multisample Processing\nGene Expression" { - shape: parallelogram -} - -multisample_adt: "Multisample Processing\nAntibody Capture" { - shape: parallelogram -} - -merge { - shape: parallelogram -} - -integration: "To Integration"{ - shape: parallelogram - style.stroke-dash: 5 -} - -input_gex_1 -> concat_gex: "unimodal\nsingle-sample" -input_gex_2 -> concat_gex: "unimodal\nsingle-sample" -input_adt_1 -> concat_adt: "unimodal\nsingle-sample" -input_adt_2 -> concat_adt: "unimodal\nsingle-sample" -input_other_1 -> concat_other: "unimodal\nsingle-sample" -input_other_2 -> concat_other: "unimodal\nsingle-sample" -concat_gex -> multisample_gex: unimodal multi-sample -concat_adt -> multisample_adt: unimodal multi-sample -concat_other -> merge: unimodal multi-sample -multisample_gex -> merge -multisample_adt -> merge -merge -> integration: "multimodal multisample" { - style.stroke-dash: 3 -} - -style: { - fill: "#FCFCFC" -} - -~~~ - - -## Multisample Gene Expression Processing -Processing multisample gene expression involved the following steps: - -1. [Normalization](../components/modules/transform/normalize_total.qmd): Normalization aims to adjust the raw counts in the dataset for variable sampling effects by scaling the observable variance to a specified range. There are different ways to transform the data, but the normalization method is to make sure each observation (cell) has a total count equal to the median of total counts over all genes for observations (cells) before normalization. -2. [Log transformation](../components/modules/transform/log1p.qmd): Calculates $X = ln(X + 1)$, which converts multiplicative relative changes to additive differences. This allows for interpreting the gene expression in terms of relative, rather than absolute, abundances of genes. -3. [Highly variable gene detection](../components/modules/filter/filter_with_hvg.qmd): Detects genes that have a large change in expression between samples. By default, OpenPipeline uses the method from Seurat [(Satija et al.)](https://doi.org/10.1038/nbt.3192). As with other "filtering" components, the `filter_with_hvg` component does not remove features, but rather annotates genes of interest by adding a boolean column to `.var`. -4. [QC metric calculations](../components/modules/qc/calculate_qc_metrics.qmd) - -~~~{.d2 height=1000px} -direction: down - -input: "Input" { - shape: page -} - -output: "Output" { - shape: page -} - -normalize: "Normalization" { - shape: parallelogram -} - -log: "Log Transformation" { - shape: parallelogram -} - -filter_with_hvg: "Highly Variable\nGene Detection" { - shape: parallelogram -} - -qc_metrics: "Calculating QC Metrics" { - shape: parallelogram -} - -input -> normalize -> log -> filter_with_hvg -> qc_metrics -> output - -style: { - fill: "#FCFCFC" -} - -~~~ - - -## Multisample Antibody Capture Processing -Processing the ADT modality for multiple samples - -~~~{.d2 pad=0} -direction: right - -input: "Input" { - shape: page -} - -output: "Output" { - shape: page -} - -normalize: "Normalization" { - shape: parallelogram -} - - -qc_metrics: "Calculating QC Metrics" { - shape: parallelogram -} - -input -> normalize -> qc_metrics -> output - -style: { - fill: "#FCFCFC" -} -~~~ - -# Integration {#sec-intergration} - -## Dimensionality Reduction {#sec-dimensionality-reduction} -scRNA-seq is a high-throughput sequencing technology that produces datasets with high dimensions in the number of cells and genes. It is true that the data should provide more information, but it also contains more noise and redudant information, making it harder to distill the usefull information. The number of genes and cells can already reduced by gene filtering, but further reduction is a necessity for downstream analysis. Dimensionality reduction projects high-dimensional data into a lower dimensional space (like taking a photo (2D) of some 3D structure). The lower dimensional representation still captures the underlying information of the data, while having fewer dimensions. - -Several dimensionality reduction methods have been developed and applied to single-cell data analysis. Two of which are being applied in OpenPipeline: - -1. [Principal Component Analysis (PCA)](../components/modules/dimred/pca.qmd): PCA reduces the dimension of a dataset by creating a new set of variables (principal components, PCs) from a linear combination of the original features in such a way that they are as uncorrelated as possible. The PCs can be ranked in the order by which they explain the largest variability in the original dataset. By keeping the top _n_ PCs, the PCs with the lowest variance are discarded to effectively reduce the dimensionality of the data without losing information. -2. [Uniform manifold approximation and projection (UMAP)](../components/modules/dimred/umap.qmd): a non-linear dimensionality technique. It constructs a high dimensional graph representation of the dataset and optimizes the low-dimensional graph representation to be structurally as similar as possible to the original graph. In a review by [Xiang et al., 2021](https://doi.org/10.3389/fgene.2021.646936) it showed the highest stability and separates best the original cell populations. -3. t-SNE is another popular non-linear, graph based dimensionality technique which is very similar to UMAP, but it has not yet been implemented in OpenPipeline. - -## Initializing integration {#sec-initializing-integration} -As will be descibed in more details [later on](#sec-integration-methods), many integration methods exist and therefore there is no single integration which is executed by default. However, there are common tasks which are run before integration either because they provide required input for many downstream integration methods or because they popular steps that would otherwise be done manually. These operations _are_ executed by default when using the "full pipeline" as part of the [initialize_integration](../components/workflows/integration/initialize_integration/initialize_integration.qmd) subworkflow. - -[PCA](../components/modules/dimred/pca.qmd) is used to reduce the dimensionality of the dataset [as described previously](#sec-dimensionality-reduction). [Find Neighbors](../components/modules/neighbors/find_neighbors.qmd) and [Leiden Clustering](../components/modules/cluster/leiden.qmd) are useful for the identification of cell types or states in the data. Here we apply a popular method to accomplish this is to first calculate a neighborhood graph on a [low dimensinonal representation](#sec-dimensionality-reduction) of the data and then cluster the data based on similarity between data points. Finally, [UMAP](../components/modules/dimred/umap.qmd) allows us to visualise the clusters by reducing the dimensionality of the data while still providing an accurate representation of the underlying cell population structure. - -~~~{.d2 pad=0} -direction: right - -input: "Input" { - shape: page -} - -output: "Output" { - shape: page -} - -pca: "PCA" { - shape: parallelogram -} - -find_neighbors: "Find\nNeighbors" { - shape: parallelogram -} - -umap: "UMAP" { - shape: parallelogram -} - -input -> pca -> find_neighbors -> umap -> output - -style: { - fill: "#FCFCFC" -} - -~~~ - -## Integration Methods {#sec-integration-methods} -Integration is the alignment of cell types across samples. There exist three different types of integration methods, based on the degree of integration across modalities: - -1. Unimodal integration across batches. For example: [scVI](../components/modules/integrate/scvi.qmd), [scanorama](../components/modules/integrate/scanorama.qmd), [harmony](../components/modules/integrate/harmonypy.qmd) -2. Multimodal integration across batches and modalities. Can be used to integrate joint-profiling data where multiple modalities are measured. For example: [totalVI](../components/modules/integrate/totalvi.qmd) -3. Mosaic integration: data integration across batches and modalities where not all cells are profiled in all modalities and it may be the case that no cells contain profiles in all integrated modalities. Mosaic integration methods have not been made available in OpenPipeline yet. An example of a tool that performs mosaic integration is StabMap. - -In either of the three cases, concatenated samples are required, and merged modalities preferred. A plethora of integration methods exist, which in turn interact with other functionality (like clustering and dimensionality reduction methods) to generate a large number of possible usecases which one pipeline cannot cover in an easy manner. Therefore, there is no single integration step that is part of a global pipeline which is executed by default. Instead, a user can choose from the integration workflows provided, and 'stack' integration methods by adding the outputs to different output slots of the MuData object. The following sections will descibe the integration workflows that are available in OpenPipeline. - -### Unimodal integration -For unimodal integration, [scVI](../components/modules/integrate/scvi.qmd), [scanorama](../components/modules/integrate/scanorama.qmd) and [harmony](../components/modules/integrate/harmonypy.qmd) have been added to the [scvi_leiden](../components/workflows/integration/scvi_leiden.qmd), [scanorama_leiden](../components/workflows/integration/scanorama_leiden.qmd), and [harmony_leiden](../components/workflows/integration/harmony_leiden.qmd) workflows respectively. After executing the integration methods themselves, [Find Neighbors](../components/modules/neighbors/find_neighbors.qmd) and [Leiden Clustering](../components/modules/cluster/leiden.qmd) are run the results of the integration as wel as [UMAP](../components/modules/dimred/umap.qmd) in order to be able to visualise the results. The functioning of these components has already been described [here](#sec-initializing-integration). - -~~~{.d2 pad=0} -direction: right - -input: "Input" { - shape: page -} - -output: "Output" { - shape: page -} - -integration: "Integration" { - shape: parallelogram -} - -integration.scvi: "scVI" { - shape: parallelogram -} - -integration.scanorama: "Scanorama" { - shape: parallelogram -} - -integration.harmony: "Harmony" { - shape: parallelogram -} - -find_neighbors: "Find\nNeighbors" { - shape: parallelogram -} - -leiden_clustering: "Leiden\nClustering" { - shape: parallelogram -} - -umap: "UMAP" { - shape: parallelogram -} - -input -> integration -> find_neighbors -> leiden_clustering -> umap -> output - -style: { - fill: "#FCFCFC" -} +:::{.callout-note} +ATAC data is carried through `process_samples` without dedicated QC or normalization — ATAC-specific processing workflows are not yet implemented. ATAC ingestion (`ingestion/cellranger_atac_count`) and a standalone QC metrics module (`calculate_atac_qc_metrics`) are available. +::: -~~~ +## Integration -### Multimodal Integration -A single multimodal integration method is currently avaiable in OpenPipeline: [totalVI](../components/modules/integrate/totalvi.qmd). It allows using information from both the gene-expression data and the antibody-capture data together to integrate the cell types. As with the other integration workflows, after running totalVI, [Find Neighbors](../components/modules/neighbors/find_neighbors.qmd), [Leiden Clustering](../components/modules/cluster/leiden.qmd) and [UMAP](../components/modules/dimred/umap.qmd) are run on the result. However in this case the three components are executed on both of the integrated modalities. +When samples come from different donors, sequencing runs, or library preparations, cells of the same biological type will cluster by their technical origin rather than their biology. **Batch correction** realigns these cells in the PCA embedding so that biological similarity — not technical source — drives the cell groupings. -~~~{.d2 pad=0} -direction: right +From the corrected embedding, a **k-nearest-neighbor graph** is built: each cell is connected to its most similar neighbours. Leiden community detection runs on this graph to assign **cluster labels**, and UMAP projects the graph into 2D for visualization. These three outputs — corrected embedding, cluster labels, UMAP — are what annotation and downstream tools consume. -input: "Input" { - shape: page -} +OpenPipelines provides one workflow per integration method. There is no single default — the right choice depends on dataset size and batch structure. See [Processing — Integration](../user_guide/processing.qmd#integration-and-batch-correction) for guidance on choosing a method. -output: "Output" { - shape: page -} +## Downstream and annotation -integration: "Integration" { - shape: parallelogram -} +After integration, the numbered Leiden clusters have no biological labels yet. **Annotation** assigns cell type identities — either by comparing marker gene expression to known references, or by transferring labels from an annotated atlas. This step is always run separately from the core pipeline and depends on the tissue and biological question. -integration.totalvi: "TotalVI" { - shape: parallelogram -} +Beyond annotation, the integrated H5MU is the starting point for differential expression, RNA velocity, cell-cell communication, and population-level analyses. Some of these are available as OpenPipelines workflows; others are external tools that consume the H5MU directly. See [Downstream](../user_guide/downstream.qmd) for details and examples. -find_neighbors: "Find\nNeighbors" { - shape: parallelogram -} +--- -leiden_clustering: "Leiden\nClustering" { - shape: parallelogram -} +## Available workflows -umap: "UMAP" { - shape: parallelogram -} +The table below lists all runnable OpenPipelines workflows and links to their component reference pages. Not every conceptual stage above has a single corresponding workflow — for instance, single-sample QC is part of `rna_singlesample` and `prot_singlesample`, and `process_samples` wraps several steps together. -input -> integration -> find_neighbors -> leiden_clustering -> umap -> output +### Ingestion -style: { - fill: "#FCFCFC" -} +| Workflow | Description | +|---|---| +| [`ingestion/demux`](../components/workflows/ingestion/demux.qmd) | BCL → FASTQ demultiplexing (bcl2fastq, BCL Convert, mkfastq) | +| [`ingestion/make_reference`](../components/workflows/ingestion/make_reference.qmd) | Build a genome reference index for mapping | +| [`ingestion/cellranger_mapping`](../components/workflows/ingestion/cellranger_mapping.qmd) | 10x GEX-only mapping with Cell Ranger | +| [`ingestion/cellranger_multi`](../components/workflows/ingestion/cellranger_multi.qmd) | 10x multiome mapping (GEX + ADT/HTO + VDJ) with Cell Ranger Multi | +| [`ingestion/bd_rhapsody`](../components/workflows/ingestion/bd_rhapsody.qmd) | BD Rhapsody mapping | +| [`ingestion/conversion`](../components/workflows/ingestion/conversion.qmd) | Convert pre-built count tables (10x HDF5, MEX, H5AD, Seurat) to H5MU | -~~~ +### Processing +| Workflow | Description | +|---|---| +| [`rna/rna_singlesample`](../components/workflows/rna/rna_singlesample.qmd) | Per-sample QC and doublet detection for RNA | +| [`prot/prot_singlesample`](../components/workflows/prot/prot_singlesample.qmd) | Per-sample QC and CLR normalization for ADT (CITE-seq) | +| [`rna/rna_multisample`](../components/workflows/rna/rna_multisample.qmd) | Concatenation, normalization, HVF, PCA for RNA | +| [`prot/prot_multisample`](../components/workflows/prot/prot_multisample.qmd) | Multi-sample normalization for ADT | +| [`multiomics/process_samples`](../components/workflows/multiomics/process_samples.qmd) | Full single-sample + multi-sample processing in one command | +| [`multiomics/process_batches`](../components/workflows/multiomics/process_batches.qmd) | Batched multi-sample processing for large cohorts | -# Putting it all together: the "Full Pipeline" {#sec-full-pipeline} +### Integration -:::{.column-screen-inset-shaded} +| Workflow | Method | Notes | +|---|---|---| +| [`integration/harmony_leiden`](../components/workflows/integration/harmony_leiden.qmd) | Harmony | Fast; good default for most datasets | +| [`integration/scvi_leiden`](../components/workflows/integration/scvi_leiden.qmd) | scVI | Deep learning; handles large datasets and complex batch effects | +| [`integration/scanorama_leiden`](../components/workflows/integration/scanorama_leiden.qmd) | Scanorama | Datasets with partial cell type overlap across batches | +| [`integration/bbknn_leiden`](../components/workflows/integration/bbknn_leiden.qmd) | BBKNN | Batch-balanced neighbor graph; no corrected embedding produced | +| [`integration/totalvi_leiden`](../components/workflows/integration/totalvi_leiden.qmd) | totalVI | Multiomics (RNA + ADT/CITE-seq); corrects both modalities jointly | -```{mermaid} -%%| label: fig-architecture -%%| fig-cap: Overview single cell processing steps in OpenPipeline. Rectangles are data objects, parallelograms are Viash modules or subworkflows. - - -flowchart TB - Raw1[/Sample 1/]:::file --> Split - Raw2[/Sample 2/]:::file --> Split2 - subgraph FullPipeline [Full Pipeline] - NoIntegration -.-> MultimodalFile[/Multisample\nMultimodal File/]:::file -.-> MultiSample - Split([Split\nmodalities]):::component --gex modality--> ProcGEX1 - Split([Split\nmodalities]):::component --prot modality--> ProcADT1 - Split([Split\nmodalities]):::component -- other--> ConcatVDJ - - Split2([Split\nmodalities]):::component --gex modality--> ProcGEX1 - Split2([Split\nmodalities]):::component --prot modality--> ProcADT1 - Split2([Split\nmodalities]):::component -- other--> ConcatVDJ - - - subgraph MultiSample [Multisample] - subgraph MultisampleRNA [Multisample RNA] - end - MultisampleRNA:::workflow - subgraph MultisampleADT [Multisample ADT] - end - MultisampleADT:::workflow - subgraph Unknown["Untreated modality (e.g. VDJ)"] - end - Unknown:::logicalgrouping - end - ConcatVDJ([Concatenate]):::component - NoIntegration[Multisample Multimodality] - - ConcatVDJ([Concatenate]):::component --> Unknown - ConcatGEX([Concatenate]):::component --> MultisampleRNA - ConcatADT([Concatenate]):::component --> MultisampleADT - - MultisampleRNA & MultisampleADT & Unknown--> Merge([Merge\nmodalities]):::component --> NoIntegration:::workflow - - end - - subgraph Wrapper - subgraph Integration[Integration] - subgraph IntegrationRNA[Integration RNA] - direction LR - hamony_leiden_rna[Harmony + Leiden]:::workflow - scvi_rna[scVI + Leiden]:::workflow - scanorama[Scanorama + Leiden]:::workflow - other[...]:::workflow - end - subgraph IntegrationProt[Integration ADT] - direction LR - hamony_leiden_prot[Harmony + Leiden]:::workflow - otherprot[...]:::workflow - end - subgraph multimodal_integration[Multimodal integration] - totalVI([totalVI]):::component - end - multimodal_integration:::logicalgrouping - IntegrationRNA:::logicalgrouping --choose from--> IntegrationProt:::logicalgrouping - NoIntegration ---> totalVI - end - Integration:::logicalgrouping - subgraph LegendBox[Legend] - direction LR - component([component]):::component - multiple_component((Multiple Components)):::component - workflow["(Sub)workflow"]:::workflow - file[/file/]:::file - Logicalgrouping[Logical grouping]:::logicalgrouping - end - LegendBox:::legend - end - Wrapper:::hide - - - NoIntegration --choose from--> IntegrationRNA - %% NoIntegration ~~~ LegendBox - - ProcGEX1[Process GEX\nSingle Sample]:::workflow --> ConcatGEX - ProcADT1[Process ADT\nSingle Sample]:::workflow --> ConcatADT - - - - style FullPipeline fill: #5cc,font-size:1.4em,color:#000; - classDef hide fill:transparent,color:transparent,stroke:transparent; - classDef legend fill:transparent; - classDef file fill: #5c5c5c,color:#fff,stroke-dasharray: 5 5; - classDef logicalgrouping fill:transparent,stroke-dasharray: 5 5; - classDef workflow fill:#ffffde,color:#000; - classDef component fill:#ececff,color:#000; +### Annotation -``` +| Workflow | Method | +|---|---| +| [`annotation/harmony_knn`](../components/workflows/annotation/harmony_knn.qmd) | KNN label transfer from a reference atlas | +| [`annotation/scvi_knn`](../components/workflows/annotation/scvi_knn.qmd) | KNN label transfer using scVI latent space | +| [`annotation/scanvi_scarches`](../components/workflows/annotation/scanvi_scarches.qmd) | scANVI/scArches label transfer | -::: +See the [Components reference](../components/) for the full argument list of each workflow and component. diff --git a/fundamentals/concepts.qmd b/fundamentals/concepts.qmd index e62d3fa9..e8a36ed8 100644 --- a/fundamentals/concepts.qmd +++ b/fundamentals/concepts.qmd @@ -22,16 +22,16 @@ This means that openpipelines must be: # Requirements -To meet these demands, the following concepts have been implemented at the core of Openpipeline: +To meet these demands, the following concepts have been implemented at the core of OpenPipelines: * 🌍 A language independent framework -* 💾 [A versitile storage solution](#sec-common-file-format) +* 💾 [A versatile storage solution](#sec-common-file-format) * 🔳 Modularity * 🔀 [A best-practice pipeline layout](architecture.qmd) * ⌛ [Versioning](../contributing/project_structure.qmd#sec-versioning) * ✅ [Automatic testing](../contributing/project_structure.qmd) * 💬 [Community input](../contributing/index.qmd) -* 📺 A graphical interace +* 📺 A graphical interface # A common file format: AnnData and MuData 💾 {#sec-common-file-format} @@ -55,23 +55,63 @@ MuData │ ├─ modality_2 (AnnData Object) ├─ .var ├─ .obs -├─ .obms +├─ .obsm ├─ .varm ├─ .uns ``` * `.mod`: an associative array of AnnData objects. Used in OpenPipelines to store the different modalities (CITE-seq, RNA abundance, ...) * `.X` and `.layers`: matrices storing the measurements with the columns being the variables measured and the rows being the observations (cells in most cases). -* `.var`: metadata for the variables (i.e. annotation for the columns of `.X` or any matrix in `.layers`). The number of rows in the .var datafame (or the length of each entry in the dictionairy) is equal to the number of columns in the measurement matrices. -* `.obs`: metadata for the observations (i.e. annotation for the rows of `.X` or any matrix in `.layers`). The number of rows in the .obs datafame (or the length of each entry in the dictionairy) is equal to the number of rows in the measurement matrices. +* `.var`: metadata for the variables (i.e. annotation for the columns of `.X` or any matrix in `.layers`). The number of rows in the .var dataframe (or the length of each entry in the dictionary) is equal to the number of columns in the measurement matrices. +* `.obs`: metadata for the observations (i.e. annotation for the rows of `.X` or any matrix in `.layers`). The number of rows in the .obs dataframe (or the length of each entry in the dictionary) is equal to the number of rows in the measurement matrices. * `varm`: the multi-dimensional variable annotation. A key-dataframe mapping where the number of rows in each dataframe is equal to the number of columns in the measurement matrices. * `obsm`: the multi-dimensional observation annotation. A key-dataframe mapping where the number of rows in each dataframe is equal to the number of rows in the measurement matrices. * `.uns`: A mapping where no restrictions are enforced on the dimensions of the data. # Modularity and a language independent framework 🔳 -TODO +OpenPipelines is built on [Viash](https://viash.io/), a framework that separates the **logic** of a component (a script in Python, R, or Bash) from its **execution environment** (a Docker container) and its **orchestration layer** (Nextflow). This separation provides two key properties. + +**Language independence.** Each component can be written in whatever language best suits the task. A normalization step might be in Python, a statistical test in R, and a file conversion in Bash. Viash wraps each script with a generated command-line interface (CLI) and injects the parsed arguments, so components are interchangeable regardless of language. + +**Modularity.** Because all components share a [common file format](#sec-common-file-format) and a generated CLI, they can be freely composed into pipelines. Adding, removing, or swapping a component requires only a change to the pipeline definition — no glue code, no format converters between steps. The same component can also be run standalone with `viash run`, tested with `viash test`, or built into a Docker image with `viash build`. + +At build time, Viash compiles each component into a Nextflow module. Workflows then chain these modules together using Nextflow DSL2, with data flowing through the pipeline as H5MU files on a shared channel. This means the pipeline scales from a laptop to an HPC cluster or cloud environment without changes to the component code. + +```{mermaid} +%%| label: fig-techstack +%%| fig-width: 7 +%%| fig-cap: "OpenPipelines technology stack. Components (code in R, Python, or Bash) are wrapped by Viash into Docker images and Nextflow modules, which are assembled into Nextflow workflows that run unchanged on local, cloud, and HPC platforms." +%%{init: {"theme":"base", "themeVariables":{"fontSize":"14px"}}}%% +graph LR + classDef input fill:#f5f5f5,stroke:#aaaaaa,color:#444 + classDef light fill:#eaf5e6,stroke:#b3d99e,color:#000 + classDef mid fill:#d5e8d4,stroke:#82b366,color:#000 + classDef core fill:#a9d69a,stroke:#5a9e57,color:#000 + + MuData["MuData"]:::input + R["R scripts"]:::light + Python["Python scripts"]:::light + Bash["Bash scripts"]:::light + Viash["Viash component"]:::core + Docker["Docker images"]:::mid + Modules["Nextflow modules"]:::mid + Nextflow["Nextflow workflows"]:::core + Local["Local execution"]:::light + AWS["AWS Batch"]:::light + GCP["Google Cloud Batch"]:::light + HPC["HPC"]:::light + More["..."]:::light + + MuData --> R & Python & Bash + R & Python & Bash --> Viash + Viash --> Docker & Modules + Docker & Modules --> Nextflow + Nextflow --> Local & AWS & GCP & HPC & More +``` # A graphical interface 📺 -TODO +[Seqera Platform](https://seqera.io/platform/) (previously Nextflow Tower) provides a web-based interface for launching, monitoring, and managing OpenPipelines workflows without using the command line. It supports multiple compute environments including AWS Batch, Google Cloud Batch, Azure Batch, and HPC schedulers. + +See the [User guide](../user_guide/running_pipelines.qmd) for instructions on how to launch workflows from the Seqera Platform UI and CLI. diff --git a/fundamentals/roadmap.qmd b/fundamentals/roadmap.qmd index ca066b90..690ca0fc 100644 --- a/fundamentals/roadmap.qmd +++ b/fundamentals/roadmap.qmd @@ -97,7 +97,7 @@ flowchart TB scanorama:::done scvi:::done totalvi:::done - scgpt_integration:::wip + scgpt_integration:::todo end intmeth:::info end @@ -106,9 +106,9 @@ flowchart TB subgraph celltype_annotation direction TB integration_method2[integration_method]:::done - celltypist:::wip + celltypist:::done scanvi:::wip - scgpt_annotation:::wip + scgpt_annotation:::todo onclass:::todo svm:::todo randomforest:::todo diff --git a/index.qmd b/index.qmd index 7c81bc5f..16be774a 100644 --- a/index.qmd +++ b/index.qmd @@ -100,47 +100,50 @@ flowchart LR %% Ingestion subgraph ingestion[Step 1: Ingestion] direction LR - 10x_ingestion[10x Ingestion]:::subwf - bd_ingestion[BD Rhapsody\nIngestion]:::subwf - own_h5mu[Own H5MU]:::subwf + 10x_ingestion[10x Ingestion]:::ingest + bd_ingestion[BD Rhapsody\nIngestion]:::ingest + own_h5mu[Own H5MU]:::ingest end - ingestion:::info + ingestion:::box %% Process samples subgraph process_samples[Step 2: Process Samples] direction LR - gex[GEX]:::subwf - atac[ATAC]:::subwf - adt[ADT]:::subwf - vdj[VDJ]:::subwf - other[Other]:::subwf + gex[GEX]:::process + atac[ATAC]:::process + adt[ADT]:::process + vdj[VDJ]:::process + other[Other]:::process end - process_samples:::info + process_samples:::box %% Integration and downstream subgraph integration[Step 3: Integration] direction LR - harmony[Harmony]:::subwf - scvi[scVI]:::subwf - scanvi[scanVI]:::subwf - etc[...]:::subwf + harmony[Harmony]:::integ + scvi[scVI]:::integ + scanvi[scanVI]:::integ + etc[...]:::integ end - integration[Integration]:::info - + integration:::box + subgraph downstream[Step 4: Downstream] direction LR - celltype_annotation[Cell Type\nAnnotation]:::subwf - markergenes[Marker Genes\nAnalysis]:::subwf - differential[Differential\nExpression]:::subwf - gene_signature_analysis[Gene Signature\nAnalysis]:::subwf - ccc[Cell-Cell\nCommunication]:::subwf + celltype_annotation[Cell Type\nAnnotation]:::down + markergenes[Marker Genes\nAnalysis]:::down + differential[Differential\nExpression]:::down + gene_signature_analysis[Gene Signature\nAnalysis]:::down + ccc[Cell-Cell\nCommunication]:::down end + downstream:::box ingestion --> process_samples --> integration --> downstream - classDef wf fill:#f0f0f0,stroke:#525252 - classDef subwf fill:#d9d9d9,stroke:#525252 - classDef info fill:#f0f0f0,stroke:#525252,stroke-dasharray: 4 4 + classDef ingest fill:#d5e8d4,stroke:#82b366,color:#000 + classDef process fill:#dae8fc,stroke:#6c8ebf,color:#000 + classDef integ fill:#ffe6cc,stroke:#d6b656,color:#000 + classDef down fill:#f8cecc,stroke:#b85450,color:#000 + classDef box fill:#f7f7f7,stroke:#cccccc,stroke-dasharray: 4 4 ``` diff --git a/more_information/code_of_conduct.qmd b/more_information/code_of_conduct.qmd index 7ceca465..67b0edd9 100644 --- a/more_information/code_of_conduct.qmd +++ b/more_information/code_of_conduct.qmd @@ -3,7 +3,7 @@ title: Code of conduct description: Our DEI values --- -We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. +We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex or gender, gender identity or expression, sexual orientation, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color or religion. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. diff --git a/more_information/faq.qmd b/more_information/faq.qmd index 632c83b1..946af3bb 100644 --- a/more_information/faq.qmd +++ b/more_information/faq.qmd @@ -3,31 +3,48 @@ title: FAQ description: Frequently Asked Questions --- - ## How can I add an external resource to my Viash component? -It is possible to add additional resources such as a file containing helper functions or other resources. All you need to do is list those files under the `functionality.resources` section of your component and refer to them in your script using `meta["resources_dir"] + "/myresource.txt"`. Please visit the Viash documentation for concrete examples on how to add [helper functions](https://viash.io/guide/component/use-helper-functions.html) and [other resources](https://viash.io/guide/component/add-resources.html) to your component. +Additional files such as helper scripts or configuration files can be included with a component by listing them under `resources` in `config.vsh.yaml`: -## What does `__merge__` do? +```yaml +resources: + - type: python_script + path: script.py + - type: file + path: helper_functions.py +``` + +Viash makes all listed resources available at runtime (including inside Docker containers) via `meta["resources_dir"]`. Reference them in your script as: + +```python +helper_path = meta["resources_dir"] + "/helper_functions.py" +``` -The `__merge__` field is used to merge another YAML into a Viash config. One of its uses is in making sure that all of the components in a task has the same API. +See the [Viash documentation](https://viash.io/guide/component/add-resources.html) for further examples. -Each task in OpenProblems contains strict definitions of the input/output file interface of its components and the file formats of those files. These interfaces are stored as YAML files in the `api` subdirectory of each task. +## What does `__merge__` do? + +`__merge__` is a Viash YAML directive that merges the contents of another YAML file into the current config at the point where it appears. In OpenPipelines it is used for author information: -:::{.content-hidden} - -2. Nextflow ([docs](https://viash.io/guide)) +```yaml +authors: + - __merge__: /src/authors/my_name.yaml + roles: [ author ] +``` -You can also change the memory and CPU utilization be editing the Nextflow labels section. Available options are `[low|med|high]` for each of `mem` and `cpu`. The corresponding resource values can be found in the `/src/wf_utils/labels.config` file. -::: +This pulls in the author's name, links, and ORCID from a central file in `src/authors/`, keeping that information consistent across all components without duplication. +## How do I view the Dockerfile that Viash generates for my component? -:::{.content-hidden} - -Docker ([docs](https://viash.io/guide)) +Pass `---dockerfile` (three dashes) to `viash run`: -**Tip #2:** You can view the dockerfile that Viash generates from the config file using the `---dockerfile` argument: -```sh -$ viash run -- ---dockerfile +```bash +viash run src/my_namespace/my_component/config.vsh.yaml -- ---dockerfile ``` -::: \ No newline at end of file + +This prints the generated Dockerfile without building it, which is useful for debugging dependency setup. + +## Where do I report bugs or request features? + +Open an issue on the [OpenPipelines GitHub repository](https://github.com/openpipelines-bio/openpipeline/issues). See the [bug reports](../user_guide/bug_reports.qmd) page for guidance on writing a useful report. diff --git a/user_guide/bug_reports.qmd b/user_guide/bug_reports.qmd index 656d6b4f..cb1b1689 100644 --- a/user_guide/bug_reports.qmd +++ b/user_guide/bug_reports.qmd @@ -5,4 +5,4 @@ order: 80 --- Issues with Openpipelines are being tracked on [Github](https://github.com/openpipelines-bio/openpipeline/issues). -In order for an issue to be fixed in a timely manner, creating a complete and reproducable is essential. +In order for an issue to be fixed in a timely manner, creating a complete and reproducible issue is essential. diff --git a/user_guide/downstream.qmd b/user_guide/downstream.qmd index f775ec63..266704b1 100644 --- a/user_guide/downstream.qmd +++ b/user_guide/downstream.qmd @@ -2,4 +2,160 @@ title: Downstream description: Celltyping and cell-cell communication order: 40 ---- \ No newline at end of file +--- + +Downstream analysis turns the integrated, clustered dataset from [processing](processing.qmd) into biological findings: what cell types are present, how they differ across conditions, how they communicate, and how they change over developmental or disease trajectories. See the [Architecture](../fundamentals/architecture.qmd) for the full pipeline overview. + +:::{.callout-note} +The analyses on this page are **not part of the core OpenPipelines processing pipeline**. They are optional next steps that depend on the biological question being asked. + +- **OP workflow wrappers** — OpenPipelines provides ready-to-run workflows for some analyses: `pseudobulk_deseq2` (wraps DESeq2), `lianapy` (wraps LIANA), `velocyto` and `scvelo` (RNA velocity). These call the underlying tools through Nextflow and produce H5MU output. +- **External tools** — The BEYOND methodology has no OP workflow wrapper; it consumes the annotated H5MU directly using its own Python package. +- **Custom analysis** — You are free to load the output H5MU into any analysis environment (Scanpy, Seurat, or custom scripts) for analyses not covered here. +::: + +## Cell type annotation + +After integration and clustering, each cluster is a group of cells with similar expression profiles — but the clusters are just numbers. Annotation assigns a biological cell type label to each cluster or individual cell. + +There are two main strategies: + +**Marker-based (manual)**: Inspect the expression of known marker genes (e.g. *CD3D* for T cells, *CD19* for B cells, *GFAP* for astrocytes) per cluster and assign labels manually. This gives full control but requires domain expertise and is not scalable. + +**Reference-based (automated)**: Transfer labels from a previously annotated reference dataset. OpenPipelines supports several automated methods: + +| Method | Component / Workflow | Notes | +|---|---|---| +| CellTypist | `annotation/celltypist` | Logistic regression; large library of pretrained immune cell models | +| SingleR | `annotate/singler` | Correlation-based; uses Bioconductor reference datasets | +| PopV | `annotate/popv` | Consensus across multiple methods; robust for diverse atlases | +| scANVI | `annotation/scanvi_scarches` | Variational autoencoder; preserves query-specific biology | +| KNN transfer | `annotation/harmony_knn`, `annotation/scvi_knn` | Direct label transfer from reference embedding | + +### Running CellTypist + +CellTypist provides pretrained models for many immune and non-immune tissues. If you do not have your own reference: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/annotation/celltypist/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id cohort_01 \ + --input cohort_01_integrated.h5mu \ + --model "Immune_All_Low.pkl" \ + --output cohort_01_annotated.h5mu \ + --publish_dir results/ +``` + +Available models are listed at [celltypist.org](https://www.celltypist.org/models). For brain or non-immune data, the `Human_AdultAged_Hippocampus.pkl` or similar tissue-specific models are available. + +### Label transfer from a custom reference + +If you have an annotated reference atlas, transfer its labels to your query: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/annotation/harmony_knn/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id query \ + --input query_integrated.h5mu \ + --reference reference_atlas.h5mu \ + --reference_obs_targets "cell_type" \ + --output query_annotated.h5mu \ + --publish_dir results/ +``` + +### Querying public atlases + +OpenPipelines can fetch cells directly from the CZ CELLxGENE Census to use as a reference: + +```bash +viash run src/query/cellxgene_census/config.vsh.yaml -- \ + --tissue "brain" \ + --cell_type "neuron" \ + --output census_neurons.h5mu +``` + +## Differential expression + +Differential expression (DE) identifies genes that are expressed at different levels across conditions (e.g. disease vs. control, treated vs. untreated) within a cell type. + +**Why pseudobulk?** A naive approach of treating each cell as an independent observation inflates the sample size (a dataset with 10 donors × 5,000 cells pretends to have 50,000 replicates). This inflated n produces false positives. The **pseudobulk** approach correctly treats biological replicates as the unit of analysis: cells are summed per donor per cell type, producing a count matrix with one row per donor — and standard bulk RNA-seq statistics (DESeq2) are applied to these aggregated counts. + +The `pseudobulk_deseq2` workflow runs both steps: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/de/pseudobulk_deseq2/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id de_ExN \ + --input cohort_01_annotated.h5mu \ + --obs_cell_type "cell_type" \ + --obs_sample_id "sample_id" \ + --obs_group "condition" \ + --reference_group "control" \ + --output de_results.h5mu \ + --publish_dir results/ +``` + +Results are stored in `.uns` of the output H5MU, one entry per cell type containing log fold changes, p-values, and adjusted p-values. + +:::{.callout-note} +You need at least 3–4 biological replicates (donors/samples) per group for DESeq2 to produce reliable estimates. Single-sample or two-sample DE should be treated with caution. +::: + +## RNA velocity + +RNA velocity estimates the *future* transcriptional state of each cell by measuring the ratio of unspliced (nascent, pre-mRNA) to spliced (mature mRNA) counts. A high unspliced/spliced ratio for a gene suggests it is being actively transcribed, giving the cell a "velocity" in gene expression space that points toward its likely future state. + +This is useful for studying: +- Differentiation trajectories (stem cells → mature cell types) +- Cell cycle progression +- Response to stimuli over time + +The OpenPipelines velocity workflow runs in two steps: + +```bash +# Step 1: extract spliced/unspliced counts from aligned BAM files +viash run src/velocity/velocyto/config.vsh.yaml -- \ + --input aligned.bam \ + --transcriptome annotation.gtf \ + --output velocity_counts.h5mu + +# Step 2: compute velocity with scVelo and project onto UMAP +viash run src/velocity/scvelo/config.vsh.yaml -- \ + --input velocity_counts.h5mu \ + --output velocity_output.h5mu +``` + +## Cell-cell communication + +Cells communicate via ligand-receptor pairs: one cell secretes a ligand (e.g. a cytokine) and a neighbouring cell expresses the matching receptor. Inferring this communication network from expression data reveals how cell types coordinate in tissue. + +The `interpret/lianapy` component uses [LIANA](https://liana-py.readthedocs.io/) to score ligand-receptor interactions across all cell type pairs: + +```bash +viash run src/interpret/lianapy/config.vsh.yaml -- \ + --input cohort_01_annotated.h5mu \ + --obs_cell_type "cell_type" \ + --methods "natmi,connectome,logfc" \ + --output lianapy_output.h5mu +``` + +LIANA aggregates results from multiple interaction databases and scoring methods, storing ranked ligand-receptor pairs in `.uns["liana_res"]`. + +## Population-level disease analysis (BEYOND) + +For disease atlas projects with multiple donors, the [BEYOND methodology](https://github.com/naomihabiblab/BEYOND_DLPFC) integrates the annotated atlas into a population-level analysis framework. OpenPipelines provides components for the BEYOND workflow: + +- `cluster/cellular_communities` — groups cell types into co-varying communities across donors +- `metadata/calculate_proportions` — computes per-donor cell type proportions +- `trajectory/pseudotime_dynamics` — fits pseudotime dynamics across donor states +- `stats/trait_associations` — associates cellular proportions with clinical traits +- `interpret/pathway_enrichment` — gene set enrichment on community-associated genes + +These components are designed to run after cell type annotation, on a multi-donor annotated atlas. + +--- + +See the [Components reference](../components/) for the full argument list of each downstream component and workflow. diff --git a/user_guide/getting_started.qmd b/user_guide/getting_started.qmd index 10ab2ea3..138c7f0d 100644 --- a/user_guide/getting_started.qmd +++ b/user_guide/getting_started.qmd @@ -4,7 +4,7 @@ description: Setting up infrastructure order: 0 --- -Depending on whether you plan to run the OpenPipelines workflows locally or in the cloud +OpenPipelines workflows run on Nextflow and Docker. This page covers what you need to install depending on whether you plan to run workflows locally or in the cloud. ## Starting workflows locally @@ -54,6 +54,6 @@ nextflow run hello -with-docker -## Using Nextflow Tower +## Using Seqera Platform -Nextflow Tower is a web-based user interface for running and monitoring Nextflow pipelines. If you are planning on using Nextflow Tower, a [compute environment](https://docs.seqera.io/platform/23.3.0/compute-envs/overview) will need to be set up. \ No newline at end of file +Seqera Platform, previously known as Nextflow Tower, is a web-based user interface for running and monitoring Nextflow pipelines. If you are planning on using Seqera Platform, a [compute environment](https://docs.seqera.io/platform/23.3.0/compute-envs/overview) will need to be set up. \ No newline at end of file diff --git a/user_guide/ingestion.qmd b/user_guide/ingestion.qmd index 5e5d4d3f..e67ed81d 100644 --- a/user_guide/ingestion.qmd +++ b/user_guide/ingestion.qmd @@ -2,4 +2,156 @@ title: Ingestion description: From sequencing to count tables order: 20 ---- \ No newline at end of file +--- + +Ingestion is the first step handled by OpenPipelines: it takes the raw output of a sequencing instrument and produces a per-sample count matrix stored as an H5MU file. See the [Architecture](../fundamentals/architecture.qmd) page for how this fits into the full pipeline. + +## What the sequencer produces + +A single-cell experiment starts with isolated cells. Each cell is barcoded (tagged with a unique short DNA sequence) and its RNA is captured and sequenced. The sequencer outputs either: + +- **FASTQ files** — plain text files containing the nucleotide reads and their quality scores. Most providers deliver these directly. +- **BCL files** — the raw binary output of Illumina sequencers. These must first be demultiplexed into per-sample FASTQ files using the `demux` workflow. + +## What ingestion does + +Ingestion aligns the short reads in the FASTQ files back to a reference genome to determine which gene each read came from. It then uses the **cell barcode** embedded in each read to assign it to a cell, and the **UMI (Unique Molecular Identifier)** to count each original RNA molecule only once — removing the amplification bias introduced by PCR. + +The result is a **count matrix**: one row per cell, one column per gene, each entry being the number of RNA molecules detected for that gene in that cell. This matrix, together with cell and gene metadata, is stored in an H5MU file. + +## Which workflow to use + +The right workflow depends on your sequencing platform and library type: + +| Starting point | Library type | Workflow | +|---|---|---| +| BCL files | Any | `ingestion/demux` → then mapping workflow | +| FASTQ | 10x GEX only | `ingestion/cellranger_mapping` | +| FASTQ | 10x multiome (GEX + VDJ + ADT/HTO) | `ingestion/cellranger_multi` | +| FASTQ | BD Rhapsody | `ingestion/bd_rhapsody` | +| FASTQ | Generic (STAR aligner) | `mapping/star_align` + `convert/from_...` | +| 10x HDF5 (`.h5`) | Pre-generated | `convert/from_10xh5_to_h5mu` | +| 10x MEX (`.mtx`) | Pre-generated | `convert/from_10xmtx_to_h5mu` | +| AnnData H5AD | Pre-generated | `convert/from_h5ad_to_h5mu` | +| Seurat RDS | Pre-generated | `convert/from_seurat_to_h5mu` | + +:::{.callout-tip} +If your sequencing provider already delivers a count matrix (e.g. Cell Ranger output), you can skip the mapping step and use one of the format converters directly. +::: + +## Reference genome + +Mapping requires a reference genome index. Build one with the `make_reference` workflow: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/make_reference/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id GRCh38 \ + --genome_fasta Homo_sapiens.GRCh38.dna.primary_assembly.fa \ + --annotation_gtf Homo_sapiens.GRCh38.111.gtf \ + --output reference_genome/ \ + --publish_dir results/ +``` + +Pre-built 10x Genomics references can be used directly. If you plan to run RNA velocity analysis downstream, spliced and unspliced counts are extracted from the BAM files produced during mapping — no special reference is required at this stage. See [RNA velocity](downstream.qmd#rna-velocity) in the downstream section. + +## Demultiplexing BCL files + +If starting from BCL files (e.g. raw Illumina output), convert them to per-sample FASTQ files first: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/demux/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id run_001 \ + --input /path/to/bcl_run/ \ + --sample_sheet SampleSheet.csv \ + --output demux_output/ \ + --publish_dir results/ +``` + +## Mapping: 10x Genomics + +### Single-library samples (GEX only) + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/cellranger_mapping/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id sample_A \ + --input "fastq/sample_A/" \ + --reference reference_genome/ \ + --output sample_A.h5mu \ + --publish_dir results/ +``` + +### Multi-library samples (GEX + ADT/HTO + VDJ) + +For CITE-seq or experiments with multiple library types captured from the same cells, Cell Ranger Multi aligns all libraries jointly: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/cellranger_multi/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id sample_A \ + --input "fastq/sample_A/" \ + --gex_reference reference_genome/ \ + --feature_reference feature_reference.csv \ + --output sample_A.h5mu \ + --publish_dir results/ +``` + +The `feature_reference.csv` describes the antibody or guide RNA sequences used in the experiment. + +## Mapping: BD Rhapsody + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/bd_rhapsody/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id sample_A \ + --input "fastq/sample_A/*.fastq.gz" \ + --reference reference.tar.gz \ + --output sample_A.h5mu \ + --publish_dir results/ +``` + +## Processing multiple samples at once + +Use `--param_list` to run ingestion in parallel across many samples. See the [Parameter lists](parameter_lists.qmd) page for supported formats (CSV, YAML, JSON). + +Example `samples.csv`: + +``` +id,input,reference +sample_A,fastq/sample_A/,reference_genome/ +sample_B,fastq/sample_B/,reference_genome/ +sample_C,fastq/sample_C/,reference_genome/ +``` + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/ingestion/cellranger_mapping/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --param_list samples.csv \ + --publish_dir results/ +``` + +## What the H5MU output contains + +After ingestion, each sample is stored as a single H5MU file. Depending on the library type, this contains one or more modalities: + +| Modality key | Content | +|---|---| +| `rna` | RNA count matrix (cells × genes) | +| `prot` | ADT count matrix — antibody-derived tags (CITE-seq protocol) | +| `vdj_b` / `vdj_t` | Immune repertoire (B/T cell receptor sequences) | +| `atac` | Chromatin accessibility (ATAC-seq peaks) | +| `gdo` | Guide RNA / CRISPR perturbation data | + +Each modality stores the raw count matrix in `.X`, along with cell metadata in `.obs` and gene/feature metadata in `.var`. At this stage no filtering has been applied — the matrix contains both real cells and empty droplets. + +The next step is [single-sample QC and processing](processing.qmd), which removes empty droplets, low-quality cells, and doublets. + +See the [Components reference](../components/) for the full argument list of each ingestion workflow. diff --git a/user_guide/parameter_lists.qmd b/user_guide/parameter_lists.qmd index e5114414..5d11b0f1 100644 --- a/user_guide/parameter_lists.qmd +++ b/user_guide/parameter_lists.qmd @@ -4,7 +4,7 @@ description: Passing multiple inputs to a workflow order: 15 --- -Using the [**Viash VDSL3 Nextflow platform**](https://viash.io/guide/nextflow_vdsl3/), an optional `--param_list` argument can be passed to OpenPipelines workflows. The `--param_list` argument enables passing multiple inputs to a workflow, resulting in a multi-event nextflow channel. +An optional `--param_list` argument can be passed to OpenPipelines workflows. The `--param_list` argument enables passing multiple inputs to a workflow, resulting in a multi-event nextflow channel. # Event-specific and global parameters @@ -35,8 +35,7 @@ will result in the following events being processed: * `id`: `foo`, `input`: `foo.txt`, `event_param`: `lorem`, `global_param`: `baz` * `id`: `bar`, `input`: `bar.txt`, `event_param`: `ipsum`, `global_param`: `baz` -Note that event-sepcific parameters defined in the `param_list` will always overwrite global parameters. For example, -running +Note that event-specific parameters defined in the `param_list` will always overwrite global parameters. For example, running ```bash nextflow run ... --param_list param_list.yaml --global_param baz @@ -48,6 +47,7 @@ with the following `param_list.yaml`: $cat param_list.yaml - id: foo input: foo.txt + global_param: lorem - id: bar input: bar.txt global_param: ipsum @@ -55,8 +55,8 @@ $cat param_list.yaml Will result in the following events being processed: -* `id`: `foo`, `input`: `foo.txt`, `event_param`: `lorem` -* `id`: `bar`, `input`: `bar.txt`, `event_param`: `ipsum` +* `id`: `foo`, `input`: `foo.txt`, `global_param`: `lorem` +* `id`: `bar`, `input`: `bar.txt`, `global_param`: `ipsum` # Other `param_list` file types @@ -101,7 +101,7 @@ nextflow run ... --param_list param_list.csv # Passing `param_list` to Nextflow's `params-file` -The `param_list` can also be passed via Nextflow's `params-file`, such that both the event-specific and global parameters are passed via a file. The following example shows how to use a `params-file` yamle file containing the `param_list`. +The `param_list` can also be passed via Nextflow's `params-file`, such that both the event-specific and global parameters are passed via a file. The following example shows how to use a `params-file` yaml file containing the `param_list`. ```bash $ cat params-file.yaml diff --git a/user_guide/processing.qmd b/user_guide/processing.qmd index fe74968e..0ada754e 100644 --- a/user_guide/processing.qmd +++ b/user_guide/processing.qmd @@ -1,5 +1,197 @@ --- -title: Processing -description: From count tables to integrated data +title: Processing & Integration +description: From count tables to a batch-corrected, clustered dataset order: 30 ---- \ No newline at end of file +--- + +Processing takes the per-sample H5MU files from [ingestion](ingestion.qmd) and produces a clean, batch-corrected dataset with Leiden clusters and a UMAP — ready for [cell type annotation and downstream analysis](downstream.qmd). See the [Architecture](../fundamentals/architecture.qmd) for the full pipeline overview. + +The recommended entry point is **`process_samples`**, which wraps per-sample QC, sample concatenation, normalization, and dimensionality reduction into a single command. For large cohorts, **`process_batches`** adds a batching layer that processes groups of samples in parallel before merging. The sections below explain what each step does and how to configure it. Integration is a separate workflow run after `process_samples`. + +## Single-sample QC and filtering + +The count matrix from ingestion contains not only real cells but also: + +- **Empty droplets** — barcodes that captured ambient RNA rather than a real cell (very low counts). +- **Dying or damaged cells** — real cells with low total counts and a high fraction of mitochondrial RNA. Mitochondrial genes are transcribed in the mitochondria and their RNA leaks into the cytoplasm when the cell membrane is damaged, inflating the mitochondrial fraction relative to nuclear RNA. +- **Doublets** — two cells captured together and sequenced as one, appearing as a cell with roughly double the expected count depth. + +The `rna_singlesample` workflow applies per-sample QC filters to remove these: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/rna/rna_singlesample/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id sample_A \ + --input sample_A.h5mu \ + --output sample_A_filtered.h5mu \ + --min_counts 500 \ + --max_counts 50000 \ + --min_genes_per_cell 200 \ + --var_name_mitochondrial_genes "mitochondrial" \ + --max_fraction_mito 0.2 \ + --publish_dir results/ +``` + +:::{.callout-tip} +QC thresholds are dataset-specific. Inspect the count distribution (`total_counts`, `n_genes_by_counts`, `pct_counts_mt`) before choosing cutoffs. A common first pass is to plot these as violin plots or scatter plots and apply thresholds at natural breakpoints. +::: + +For CITE-seq data (RNA + ADT), run `prot_singlesample` alongside `rna_singlesample` to apply CLR (Centered Log-Ratio) normalization to the ADT (`prot`) modality. + +Process all samples in parallel with a parameter list: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/rna/rna_singlesample/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --param_list samples.csv \ + --min_counts 500 \ + --max_counts 50000 \ + --min_genes_per_cell 200 \ + --publish_dir results/ +``` + +## Multi-sample normalization and dimensionality reduction + +After filtering, samples are concatenated and normalized together. The `rna_multisample` workflow performs: + +1. **Concatenation** — all per-sample H5MU files are joined into one object. +2. **Normalization** — each cell is scaled to a fixed total count (e.g. 10,000), making cells comparable regardless of sequencing depth. A log1p transformation is then applied to stabilize variance (raw counts follow a negative binomial distribution where variance grows with the mean). +3. **Highly variable feature (HVF) selection** — most genes do not vary meaningfully across cells. Focusing on the 2,000–3,000 most variable genes retains biological signal while reducing noise and computation time. +4. **PCA** — Principal Component Analysis compresses the high-dimensional gene expression space into a smaller set of components capturing the major axes of variation. Subsequent steps work in this reduced space. + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/rna/rna_multisample/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id cohort_01 \ + --input "results/sample_*_filtered.h5mu" \ + --output cohort_01_multisample.h5mu \ + --publish_dir results/ +``` + +### Running the full pipeline in one command + +`process_samples` wraps `rna_singlesample`, `prot_singlesample`, concatenation, and `rna_multisample` into a single workflow, removing the need to chain steps manually: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/multiomics/process_samples/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --param_list samples.csv \ + --publish_dir results/ +``` + +For large cohorts, `process_batches` splits samples into groups, processes each group through multisample normalization in parallel, then merges results — reducing peak memory usage. Use `process_batches` directly when you already have per-sample filtered files and want to run the multisample steps at scale. See [Parameter lists](parameter_lists.qmd) for the `samples.csv` format. + +--- + +:::{.callout-important} +## Integration is a separate workflow + +The steps below describe a **separate Nextflow workflow** that takes the `process_samples` output as input. Integration is not part of `process_samples` — it must be run independently once per cohort. +::: + +## Integration and batch correction + +Samples from different donors, sequencing runs, or library preparations will cluster by technical origin rather than biology if not corrected. **Batch correction** aligns the cell type clusters across batches without erasing real biological variation. + +### The dimensionality reduction chain + +Integration workflows run a fixed sequence of steps on top of the PCA produced by `process_samples`: + +1. **Batch correction** — adjusts the PCA embedding so that cells from different batches with the same biological identity overlap. Harmony operates directly in PCA space; scVI and totalVI learn a corrected latent space using a variational autoencoder. +2. **Neighbor graph** — constructs a k-nearest-neighbor (kNN) graph in the corrected embedding space. Each cell is connected to its most similar neighbours. This graph is the basis for both clustering and UMAP. +3. **Leiden clustering** — applies community detection to the neighbor graph. Cells that are densely connected form a cluster. The `--leiden_resolution` parameter controls granularity: higher values produce more, smaller clusters. +4. **UMAP** — projects the neighbor graph into 2D for visualization. UMAP preserves local neighbourhood structure, so nearby cells in the plot are genuinely similar, but distances between clusters are not quantitatively meaningful. + +The corrected embedding (`obsm["X_pca_integrated"]`) is what downstream tools like label transfer and trajectory analysis use — not the UMAP coordinates. + +OpenPipelines provides separate integration workflows — one per method — each running this full chain: + +| Workflow | Method | When to use | +|---|---|---| +| `integration/harmony_leiden` | Harmony | Fast, good default for most datasets | +| `integration/scvi_leiden` | scVI | Deep learning; handles complex batch structures, large datasets | +| `integration/scanorama_leiden` | Scanorama | Datasets with partial cell type overlap across batches | +| `integration/bbknn_leiden` | BBKNN | Batch-balanced neighbor graph; no corrected embedding | +| `integration/totalvi_leiden` | totalVI | Multiomics (RNA + ADT / CITE-seq); corrects both modalities jointly | + +Example using Harmony (good starting point): + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/integration/harmony_leiden/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id cohort_01 \ + --input cohort_01_multisample.h5mu \ + --obs_covariates "sample_id" \ + --leiden_resolution 1.0 \ + --output cohort_01_integrated.h5mu \ + --publish_dir results/ +``` + +:::{.callout-note} +`--obs_covariates` must match an existing `.obs` column that identifies the technical source of variation — typically the sample ID, batch, or donor. For multi-donor studies, using donor ID as the covariate is common. +::: + +Example using scVI for a large or complex dataset: + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/integration/scvi_leiden/main.nf \ + -revision 4.0.3 -latest -profile docker \ + --id cohort_01 \ + --input cohort_01_multisample.h5mu \ + --obs_batch "sample_id" \ + --leiden_resolution 1.0 \ + --output cohort_01_integrated.h5mu \ + --publish_dir results/ +``` + +### Choosing an integration method + +There is no universally best method. A practical guide: + +- Start with **Harmony** — it is fast, interpretable, and works well for most datasets. +- Use **scVI** when you have many samples (>20), strong batch effects, or need a probabilistic latent space for downstream analysis. +- Use **totalVI** when you have matched RNA + ADT (CITE-seq) and want to correct both modalities jointly. +- Use **Scanorama** when batches have different cell type compositions (e.g. different tissues). +- **BBKNN** modifies only the neighbor graph and does not produce a corrected embedding — it is lighter but less flexible for downstream uses. + +## Using a parameters file + +For reproducibility, store all parameters in YAML: + +```yaml +# params.yaml +id: cohort_01 +input: cohort_01_multisample.h5mu +output: cohort_01_integrated.h5mu +obs_covariates: "sample_id" +leiden_resolution: 1.0 +``` + +```bash +nextflow run openpipelines-bio/openpipeline \ + -main-script target/nextflow/workflows/integration/harmony_leiden/main.nf \ + -revision 4.0.3 -latest -profile docker \ + -params-file params.yaml \ + --publish_dir results/ +``` + +## Output + +The integrated H5MU contains: + +| Slot | Content | +|---|---| +| `.obsm["X_pca_integrated"]` | Batch-corrected cell embedding | +| `.obs["leiden"]` | Leiden cluster labels | +| `.obsm["X_umap"]` | UMAP coordinates for visualization | +| `.obsp["distances"]` / `.obsp["connectivities"]` | Neighbor graph | + +Clusters are numbered — they have no biological meaning yet. The next step, [cell type annotation](downstream.qmd), assigns biological labels. + +See the [Components reference](../components/) for the full argument list of each workflow. diff --git a/user_guide/running_pipelines.qmd b/user_guide/running_pipelines.qmd index 7e251b05..2a8312ff 100644 --- a/user_guide/running_pipelines.qmd +++ b/user_guide/running_pipelines.qmd @@ -1,10 +1,10 @@ --- title: Running pipelines -description: Run a pipeline from CLI or Nextflow Tower +description: Run a pipeline from CLI or Seqera Platform order: 10 --- -Dependening on whether you want to run a workflow locally or on cloud infrastructure, using Nextflow Tower or not, you will need to use different commands. +Depending on whether you want to run a workflow locally or on cloud infrastructure, using Seqera Platform (Nextflow Tower) or not, you will need to use different commands. ## Run locally from the CLI @@ -12,13 +12,14 @@ You can run a workflow from the command line using the following command: ```bash nextflow run openpipelines-bio/openpipeline \ - -main-script target/nextflow/workflows/integration/multimodal_integration/main.nf \ - -revision 0.12.1 \ + -main-script target/nextflow/workflows/integration/harmony_leiden/main.nf \ + -revision 4.0.3 \ -latest \ -profile docker \ - --publish_dir foo/ \ - --input "bar" \ - --output "test.txt" + --publish_dir results/ \ + --id my_sample \ + --input input.h5mu \ + --output output.h5mu ``` Doing so will run the workflow **locally** using a Docker container. @@ -29,18 +30,20 @@ You can use a similar command to run the workflow on cloud infrastructure, such ```bash nextflow run openpipelines-bio/openpipeline \ - -main-script target/nextflow/workflows/integration/multimodal_integration/main.nf \ - -revision 0.12.1 \ + -main-script target/nextflow/workflows/integration/harmony_leiden/main.nf \ + -revision 4.0.3 \ -latest \ - --publish_dir foo/ \ - --input "bar" \ - --output "test.txt" \ + --publish_dir results/ \ + --id my_sample \ + --input input.h5mu \ + --output output.h5mu \ -c configs/my_hpc.config ``` -## Using the Nextflow Tower CLI +## Using the Seqera Platform CLI + +If you have access to a Seqera Platform (previously Nextflow Tower) instance in which a Compute Environment has already been set up, you can run a workflow from the Seqera CLI. The command is very similar to the command to run a workflow from the CLI, but you need to: -If you have access to a Nextflow Tower instance in which a Compute Environment has already been set up, you can run a workflow from the Tower CLI. The command is very similar to the command to run a workflow from the CLI, but you need to: * Use `tw launch` instead of `nextflow run` * Specify the workspace ID and compute environment ID * Rename arguments: `-revision` to `--revision`, `-latest` to `--pull-latest`, `-main-script` to `--main-script`, `-c` to `--config` @@ -50,18 +53,18 @@ Example: ```bash tw launch openpipelines-bio/openpipeline \ - --revision 0.12.1 \ + --revision 4.0.3 \ --pull-latest \ - --main-script target/nextflow/workflows/integration/multimodal_integration/main.nf \ + --main-script target/nextflow/workflows/integration/harmony_leiden/main.nf \ --workspace \ --compute-env \ --params-file params.yaml \ --config configs/my_hpc.config ``` -## Using the Nextflow Tower Web UI +## Using the Seqera Platform Web UI -If you have access to a Nextflow Tower instance in which a Compute Environment has already been set up, you can run a workflow from the Tower UI. To do so, go to the "Launchpad" and click on the button "launch a run without configuration". +If you have access to a Seqera Platform (previously Nextflow Tower) instance in which a Compute Environment has already been set up, you can run a workflow from the web UI. To do so, go to the "Launchpad" and click on the button "launch a run without configuration". ![](./images/launchpad.png) @@ -69,6 +72,6 @@ Next, fill in the required fields and click on "Launch run". * **Compute environment**: Select the compute environment you want to run the workflow on. * **Pipeline to launch**: Fill in `openpipelines-bio/openpipeline`. -* **Revision number**: The release number of the pipeline you want to run, e.g. `0.12.1`. You can find the release number on the [GitHub releases page](https://github.com/openpipelines-bio/openpipeline/releases). +* **Revision number**: The release number of the pipeline you want to run, e.g. `4.0.3`. You can find the release number on the [GitHub releases page](https://github.com/openpipelines-bio/openpipeline/releases). * **Work directory**: The bucket path where the scratch data is stored. * **Pipeline parameters**: The YAML or JSON of the parameters that are passed to the pipeline. See the [Components](../components/) page for more information about the parameters of each pipeline.