Workflows in nextflow

[sjfleming]

This might be a much much better way than #2 to use use google batch (and local). (Could do aws in future too.)

So far nextflow is seeming fantastic.

The whole CAS pca training can be run by doing this

nextflow run cellarium/workflows/nextflow/pca_split_workflow.nf -profile gcp --train_dataset_dir gs://cellarium-nexus-file-system-3293a8/pipeline/data-extracts/20260403_cas_pca_model_10x/extract_files --predict_dataset_dir gs://cellarium-nexus-file-system-3293a8/pipeline/data-extracts/20260403_cas_pca_vsindex_10x/extract_files --outdir gs://cellarium-dev-central/workflows/nextflow_cas_pca --var_names_key feature_id --n_top_genes 4000 --n_components 64

Install is just

conda create -n nextflow bioconda::nextflow

You have to run it on a server somewhere with a tmux session, because the place you run it from must remain on.