Reconstructed documentation proposal

.gitignore

@@ -14,4 +14,5 @@ _site
 *_files/
 site_libs
 
-/.luarc.json
+/.luarc.json
+**/*.quarto_ipynb

CHANGELOG.md

@@ -4,21 +4,27 @@
 
 * Add descriptions to all pages and add listings to index pages.
 
-* Update documentation on creating components for developers.
+* Update documentation on creating components for developers, including updated config examples reflecting Viash 0.9.x syntax (`argument_groups`, `engines`, `runners`).
 
-* Update getting started page for developers
+* Add guide for creating new pipelines, covering workflow structure, config layout, `main.nf` conventions, and testing.
+
+* Expand user guide: ingestion, processing, and downstream analysis pages now contain full content covering workflow inputs, outputs, and usage patterns.
+
+* Rewrite architecture page to provide a concise, diagram-light overview of how OpenPipelines fits into a single-cell project.
+
+* Update getting started page for developers.
 
 * Update project structure.
 
-* Update information on running tests.
+* Update information on running tests, including how to generate local test data without downloading from S3.
 
-* Update "More information" pages
+* Update "More information" pages.
 
-* Write getting started page for user guide
+* Write getting started page for user guide.
 
-* Document how to run workflows
+* Document how to run workflows.
 
-* Document parameter lists
+* Document parameter lists.
 
 # OpenPipelines.bio v0.12.1
 

contributing/creating_components.qmd

@@ -6,7 +6,7 @@ order: 20
 
 ## A common file format
 
-One of the core principals of OpenPipelines is to use [MuData](https://mudata.readthedocs.io/) as a common data format troughout the whole pipeline. See [the concepts page](/fundamentals/concepts.qmd#sec-common-file-format) for more information on openpipelines uses MuData to store single-cell data.
+One of the core principals of OpenPipelines is to use [MuData](https://mudata.readthedocs.io/) as a common data format throughout the whole pipeline. See [the concepts page](/fundamentals/concepts.qmd#sec-common-file-format) for more information on openpipelines uses MuData to store single-cell data.
 
 ## Component location
 As discussed in [the project structure](project_structure.qmd#sec-project-structure), components in the repository are stored within `src`. Additionally, components are grouped into namespaces, according to a common functionality. An example of such a namespace is the dimensionality reduction namespace (`dimred`), of which the components `pca` and `umap` are members. This means that within `src`, the namespace folders can be found that stores the components that belong to these namespaces. 
@@ -27,31 +27,46 @@ A component consists of one or more scripts that provide the functionality of th
 ## The config
 
 ```yaml
-functionality:
-  name: "my_component"
-  namespace: "my_namespace"
-  description: "My new custom component"
-  authors:
-    - __merge__: ../../authors/my_name.yaml
-      roles: [ author ]
-  arguments:
-    - name: "--output"
-      type: file
-      example: "output_file.h5mu"
-      description: "Location were the output file should be written to."
-      direction: "output"
-  resources:
-    - type: python_script
-      path: script.py
-platforms:
+name: "my_component"
+namespace: "my_namespace"
+description: "My new custom component"
+authors:
+  - __merge__: ../../authors/my_name.yaml
+    roles: [ author ]
+
+argument_groups:
+  - name: "Inputs"
+    arguments:
+      - name: "--input"
+        type: file
+        required: true
+        description: "Input H5MU file."
+        example: "input.h5mu"
+  - name: "Outputs"
+    arguments:
+      - name: "--output"
+        type: file
+        direction: output
+        description: "Location where the output file should be written to."
+        example: "output.h5mu"
+
+resources:
+  - type: python_script
+    path: script.py
+
+engines:
   - type: docker
     image: python:3.11
     setup:
       - type: python
-        packages: mudata~=0.2.3
+        packages:
+          - mudata~=0.3
+
+runners:
   - type: nextflow
     directives:
       label: [highcpu, midmem]
+  - type: executable
 ```
 ### Basic information
 
@@ -66,16 +81,16 @@ Basic information checklist:
 
 ### Arguments and argument groups
 
-If you component requires arguments, they should be defined in `arguments` or `argument_groups`. Try tro group individual arguments into `argument_groups` when the number of arguments become too larg  (10 or more as a rule of thumb). 
+If your component requires arguments, they should be defined in `argument_groups`. Try to group individual arguments into `argument_groups` when the number of arguments become too large (10 or more as a rule of thumb). 
 
 Argument checklist:
 
  - Add a description and name
- - Each argument should have the appropriate type.
+ - Each argument should have the appropriate type
  - Input and output files should be of type `file` instead of `string` and use the appropriate `direction:` 
  - If possible: add an example
  - If the argument can accept multiple values, add `multiple: true`
- - If the possible input for an argument is limited to certain set of values, use `choices:`
+ - If the possible input for an argument is limited to a certain set of values, use `choices:`
 
 ### (Test)resources
 
@@ -84,49 +99,240 @@ Resources define files that are required for a component to perform its function
 There is a difference between defining `resources` and `test_resources`. While resources are required for a component to function, `test_resources` only need to be included when testing the component (with for example `viash test`) in addition to the regular resources. Having a look at the example above, resources are defined using the `resources:` property. It takes a list of multiple files or folders.
 
 In openpipelines, it was decided to not use a service like `git lfs` to include large resources into the repository. Instead, if large resources are required, there are two possibilities:
+
   * Large resources required for _testing_ are to uploaded into an `s3` bucket that is synced automatically before running tests (both locally and on github). Please ping a maintainer when you open a PR and ask them to upload the files for you.
-  * Other large resources that are not needed for testing can be considered as input. This means that an argument of `type: file` needs to be created. The downside of this method is that viash is not able to natively support remote files f
+  * Other large resources that are not needed for testing can be considered as input. This means that an argument of `type: file` needs to be created. The downside of this method is that viash is not able to natively support remote files.
 
 Resources checklist:
+
   - Script resources are located next to the config and added to the config with the correct type (`python_script`, `r_script`, ...)
-  - Small resources (<50MB) that are not scripts can also be checked in into the repo, next to the 
+  - Small resources (<50MB) that are not scripts can also be checked into the repo, next to the `config.vsh.yaml`.
 
 ### The script file
 
-TODO
+The script file contains the actual implementation of the component. Viash injects the parsed CLI arguments into the script using a `## VIASH START` / `## VIASH END` marker block. During development, this block holds placeholder values so the script can be run directly in an IDE or REPL. At build time, Viash replaces the block with generated code that reads the actual CLI arguments.
+
+::: {.panel-tabset}
+
+# Python
+
+```python
+## VIASH START
+par = {
+    "input": "input.h5mu",
+    "output": "output.h5mu",
+    "modality": "rna",
+    "min_counts": 200,
+}
+meta = {
+    "resources_dir": ".",
+    "temp_dir": "/tmp",
+    "cpus": 1,
+    "memory_gb": 4,
+}
+## VIASH END
+
+import mudata as mu
+
+mdata = mu.read_h5mu(par["input"])
+adata = mdata.mod[par["modality"]]
+
+# ... component logic ...
+
+mdata.write_h5mu(par["output"])
+```
+
+# R
+
+```r
+## VIASH START
+par <- list(
+  input = "input.h5mu",
+  output = "output.h5mu",
+  modality = "rna"
+)
+meta <- list(
+  resources_dir = ".",
+  temp_dir = "/tmp",
+  cpus = 1L,
+  memory_gb = 4L
+)
+## VIASH END
+
+library(MuData)
+
+mdata <- readH5MU(par$input)
+
+# ... component logic ...
+
+writeH5MU(mdata, par$output)
+```
+
+# Bash
+
+```bash
+## VIASH START
+par_input="input.h5mu"
+par_output="output.h5mu"
+par_modality="rna"
+meta_cpus=1
+meta_memory_gb=4
+## VIASH END
+
+set -eo pipefail
+
+# unset boolean parameters before use
+[[ "$par_some_flag" == "false" ]] && unset par_some_flag
+
+# build argument array
+cmd_args=(
+  --input "$par_input"
+  --output "$par_output"
+  --modality "$par_modality"
+)
+[[ -n "$par_some_flag" ]] && cmd_args+=(--flag)
+
+mytool "${cmd_args[@]}"
+```
+
+:::
+
+Script checklist:
+
+  - The `## VIASH START` / `## VIASH END` block is present with realistic placeholder values
+  - Use `meta_cpus` and `meta_memory_gb` for resource-aware parallelism — never hardcode thread counts
+  - Bash scripts: use `set -eo pipefail` after `## VIASH END` and build CLI arguments as arrays
 
 ### Author information
 
-TODO
+Author information is centralised in `src/authors/`. Each contributor has a YAML file with their profile, which is merged into individual component configs using Viash's `__merge__` property. This keeps author information consistent across all components without duplication.
+
+To add yourself as an author:
+
+1. Create `src/authors/your_name.yaml`:
+
+```yaml
+name: Your Name
+info:
+  links:
+    - type: github
+      url: https://github.com/your_handle
+    - type: orcid
+      url: https://orcid.org/0000-0000-0000-0000
+```
+
+2. Reference it in your component's `config.vsh.yaml`:
+
+```yaml
+authors:
+  - __merge__: ../../authors/your_name.yaml
+    roles: [ author ]
+```
+
+Valid roles are `author` and `maintainer`. The `maintainer` role indicates who is responsible for keeping the component up to date.
 
 ## Adding dependencies
 
-TODO
+Dependencies are declared in the `engines` section of `config.vsh.yaml`. Viash uses this information to build the Docker image for the component. Always prefer installing packages through a package manager over building from source.
+
+::: {.panel-tabset}
+
+# Python (pip)
+
+```yaml
+engines:
+  - type: docker
+    image: python:3.11
+    setup:
+      - type: python
+        packages:
+          - mudata~=0.3
+          - scanpy>=1.9
+          - numpy
+```
+
+# Python (conda / BioContainers)
+
+BioContainers images from `quay.io/biocontainers` are preferred for bioinformatics tools because they pin exact versions including the build string (e.g. `tool:2.0.3--h5b5514e_1`), improving reproducibility:
+
+```yaml
+engines:
+  - type: docker
+    image: quay.io/biocontainers/scanpy:1.9.3--pyhdfd78af_0
+    setup:
+      - type: python
+        packages:
+          - mudata~=0.3
+```
+
+# R
+
+```yaml
+engines:
+  - type: docker
+    image: eddelbuettel/r2u:22.04
+    setup:
+      - type: r
+        cran:
+          - MuData
+          - Seurat
+        bioc:
+          - SingleR
+          - scran
+```
+
+# System packages (apt)
+
+```yaml
+engines:
+  - type: docker
+    image: ubuntu:22.04
+    setup:
+      - type: apt
+        packages:
+          - samtools
+          - libhdf5-dev
+      - type: python
+        packages:
+          - mudata
+```
+
+:::
+
+After editing the setup, rebuild the Docker image to verify it installs correctly:
+
+```bash
+viash build src/my_namespace/my_component/config.vsh.yaml \
+  --engine docker --setup cb
+```
+
+The `cb` (cachedbuild) flag reuses cached layers where possible.
 
 ## Building components from their source
-When running or [testing individual components](#running-component-unittests), it is not necessary to execute an extra command to run the build step, `viash test` and `viash run` will build the component on the fly. However, before integrating components into a pipeline, you will need to build the components. More specifically, openpipelines uses Nextflow to combine components into pipelines, so we need to have at least the components build for `nextflow` platform as target. The easiest method to build the components is to use:
+When running or [testing individual components](running_tests.qmd), it is not necessary to execute an extra command to run the build step, `viash test` and `viash run` will build the component on the fly. However, before integrating components into a pipeline, you will need to build the components. More specifically, openpipelines uses Nextflow to combine components into pipelines, so we need to have at least the components build for `nextflow` platform as target. The easiest method to build the components is to use:
 
 ```bash
-viash ns build --parallel --setup cachedbuild
+viash ns build --setup cb
 ```
-After using `viash ns build`, the target folder will be populated with three subfolders, corresponding to the build platforms that viash supports: `native`, `docker` and `nextflow`. 
+After using `viash ns build`, the `target/` folder is populated with build outputs for each declared engine and runner (typically `docker` and `nextflow`).
 
-Building an individual component can still be useful, for example when debugging a component for which the build fails or if you want to create a standalone executable for a component to execute it without the need to use `viash`. To build an individual component, `viash build` can be used. Note that the default build directory of this viash base command is `output`, which is not the location where build components will be imported from when integrating them in pipelines. Using the `--output` argument, you can set it to any directory you want, for example:
+Building an individual component can be useful when debugging a build failure or creating a standalone executable. Note that the default output directory is `output/`, not `target/` — use `--output` to set the correct location:
 
 ```bash
-viash build src/filter/do_filter/config.vsh.yaml -o target/native/filter/do_filter/ -p native
+viash build src/filter/do_filter/config.vsh.yaml \
+  -o target/executable/filter/do_filter/ --engine native
 ```
 
 ## Containerization
 One of the key benefits of using Viash is that containers can be created that gather dependencies per component,
-which avoids building one container that has to encorporate all dependencies for a pipeline together. The containers for a single component can be reduced in size, defining the minimal requirements to run the component. That being said, building containers from scratch can be labour intensive and error prone, with base containers from reputable publishers often benefiting from improved reliability and security. Hence, a balance has to be made between reducing the container's size and adding many dependencies to a small base container.
+which avoids building one container that has to incorporate all dependencies for a pipeline together. The containers for a single component can be reduced in size, defining the minimal requirements to run the component. That being said, building containers from scratch can be labour intensive and error prone, with base containers from reputable publishers often benefiting from improved reliability and security. Hence, a balance has to be made between reducing the container's size and adding many dependencies to a small base container.
 
 The preferred containerization setup in OpenPipelines uses the following guidelines:
 
 * Choose a base container from a reputable source and use its latest version
 * Do not use base containers that have not been updated in a while
 * Use package managers to install dependencies as much as possible
-* Avoid building depdencies from source.
+* Avoid building dependencies from source.
 
 Examples of base containers that are currently being used are: