docs

System of Pathology Agents for Research and Knowledge

System of Pathology Agents for Research and Knowledge (SPARK) is a novel agentic AI framework that uses language as a universal interface to autonomously reason, generate biologically driven concepts for tumor analysis, implement them as tools, and manipulate complex pathology data without additional training.

Journal Link | Cite

© Tolkach Lab

Content

• Modular Presentation

• Selection of LLM

• Prompt Design

• Installation

• Deployment of LLM

• Quick Start

  • <1> Generative pipeline
  • <2> Analytical pipeline
  • <3> Prognostic Pipeline

• More Setup

• Cite

What is SPARK? - Modular presentation

SPARK is a multi-agent AI workflow that utilizes CrewAI, a multi-agent framework, and large language models (LLMs) to generate biological concepts and translate them into analytical tools. The workflow consists of multiple modules that are modular and flexible—each can be easily modified or replaced using natural language according to user needs, allowing adaptation to many applications.

SPARK can be roughly split into three main modules (crews):

1. Generative Pipeline This pipeline transforms AI- or human-initiated concepts into structured biological hypotheses and proposes corresponding methodological implementations. Each idea is reviewed for guideline adherence and checked for duplicates to ensure novelty. Refined ideas are then expanded into detailed, stepwise analytical instructions and translated into executable code snippets. A final verification step confirms that the generated code is logical, meaningful, and produces valid outputs.

For detailed explanations and examples of each module, see the Generative_Pipelines file.

Three idea generation methods (B, C, and D) are available, each targeting a different dataset type and biological question but following the same principle: Generate one well-defined hypothesis that can be quantified from WSI- or GeoJSON-derived data to predict clinical outcomes.

• Idea_generation_B — Predicting Survival / Tumor Progression

  Goal: Generate hypotheses describing parameters that can predict patient survival or tumor progression.

  Input data:

  • H&E-stained whole-slide images (WSI)
  • Tissue segmentation masks (epithelial tumor and stroma)
  • GeoJSON file with 7 single-cell classes

  Focus: Parameter-level idea specifying the biological hypothesis.

• Idea_generation_C — Predicting Metastatic Spread

  Goal: Generate hypotheses identifying features that distinguish metastatic vs. non-metastatic tumors.

  Input data:

  • Same as idea_generation_B (H&E + segmentation + GeoJSON with 7 cell types)

  Focus: Microenvironmental and cellular mechanisms that promote metastasis, such as: Immune cell exclusion or clustering, fibroblast density gradients, spatial organization patterns around tumor nests.

• Idea_generation_D — Multiplexed Tissue Microarray (TMA) Analysis

  Goal: Generate hypotheses for survival or progression prediction using multiplexed TMA data

  Input data:

  • Tissue microarray (TMA) regions
  • GeoJSON data containing up to 14 distinct cell and structure types, including immune subtypes, fibroblast subsets, and vessel structures.

  Focus: Use marker-defined subpopulations and tissue structures to propose parameters that describe the tumor microenvironment and its prognostic role.

The complexity level (basic, intermediate, advanced)

2. Analytical Pipeline This pipeline demonstrates how users can apply the generated code snippets to analyze Whole Slide Image (WSI) data. In our experiments, we used the WSI, a tissue segmentation mask (covering tumor and stroma compartments), and a GeoJSON file containing cell detection and classification data as inputs. The generated code snippets were then used to analyze single cells in these slides. Two evluation regimes are available to address different biological problems. Please see Analytical pipeline for further work-on details.

Note: The invasion front and tumor center detection module works similarly but further labels image patches as belonging to the tumor core or invasion front using detailed multi-class tissue segmentation masks.

For detailed explanations and examples of each module, see the Analytical_Pipeline and Analytical_Pipeline_invasion_tumor_center files.

3. Prognostic Pipeline: This pipeline evaluates the prognostic significance of features derived from the analytical pipeline by integrating them with clinical and survival data. It aggregates per-slide measurements into a unified patient-level database, computing summary statistics for each verified parameter. Using this database, it performs comprehensive survival analyses—including univariate and multivariate Cox regression, Kaplan–Meier curve generation, and correlation analyses with clinical variables—to identify morphological or spatial tissue features significantly associated with patient outcomes and potential prognostic biomarkers.

Before you start - selection of LLM and serving platforms

The generative pipline utilizes LLMs for idea generation, review, convertion to codes, and related tasks. The system supports both locally deployed and API-based LLMs. We recommend vllm for deployment - this is the platform used in the publicationbut - but other LLM-serving platforms such as Ollama are also supported. Follow the guidelines on each project's official documentation for serving models and running inference. In our work we used ChatGPT-o1 and claude-3-5-sonnet for idea manipulation and coding tasks respectively. You may choose any publicly available LLMs.

Prompt Design and Configuration

CrewAI relies on LLMs, so effective prompting is essential to achieve the desired results. To optimize performance, users should design and customize prompts carefully. Agent descriptions are defined in ./config/agents.yaml (CrewAI uses YAML as its configuration format), while task-specific instructions are located in ./config/tasks.yaml. We strongly recommend that users create their own prompts in these files to tailor the system to their specific research goals. However, you may also use the provided configurations to ensure reproducibility in terms of input format, output format, and analytical focus.

Installation:

Please follow the guidelines from Installation guide for the installation of all needed python packages.

(Optional) Deployment of LLM

This section is only relevant for users deploying local LLMs. If you plan to use API keys to access LLMs directly, you can skip this part.

Ensure the spark environment is activated following Installation guide.

Since we use vllm as the LLM-serving platform, deploying any public LLM can be done easily by following the instructions from vllm’s quickstart guide.

For example, to deploy DeepSeek-R1, you can use the following command (adjust the entries as needed)::

vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B   
  --tensor-parallel-size 4   #Number of GPU cards
  --host 0.0.0.0   #Define you own IP address, 0.0.0.0 returns the IP address of your work-station by default
  --port 8181 #Choose a port for the entry

We used this configuration in 4x NVIDIA A100, 4x ~86 GB RAM setting. Please consider adjusting the configurations according to the hardware in your work station.

Quick Start

The following sections explain how to generate AI-explored or human-initiated biologically driven ideas, convert them into analyzable Python code snippets, evaluate single-cell data at the slide level, aggregate results at the patient level, and perform prognostic analysis. These pipelines are highly versatile, with many adjustable parameters that can be tailored to specific research goals. The examples below demonstrate a quick workflow for reproducing the same process described in the publication using your own datasets. More detailed information about configurations and parameter settings can be found in General Setup.

<1> Generative pipeline

!!IMPORTANT!! Before starting the steps, please define which use case you want to focus on. A short summary of each use case can be found in the What is SPARK? - Modular presentation section. Replace X in the tutorial with the corresponding use case letter. Detailed prompts and agent instructions are provided agents.yaml and tasks.yaml.

Then, set the working directory:

cd ./Generative_and_Prognostic_Pipeline

The idea generation module can be split into three major parts:

• Step 1a. Idea Generation via AI

  To let the LLM to generate biological-driven ideas, please consider the following steps.

  • Step 1a.1 Open config_IG_X.json: change the parts to the LLM model name in LLM_IGA, LLM_IRA and LLM_DDA to the name of the LLM documented in HuggingFace (including the prefix) or follow the instruction on their official websites.
  • Step 1a.2 Open agent_task_init.py: replace base_url according to SPRAK Idea Generation General Setup (Task initiation) in initialize_LLM. (Please keep provider="openai" no matter which model you use).
  • Step 1a.3 (Add API key if needed in main_IG_X_UC#.py). Run main_IG_X_UC#.py.

• Step 1b. Idea Generation via human-initiated ideas

  SPARK also allows ideas thought by human beings. And it is able to turn them formatted ideas. Several considerations need to be taken before starting the process:

  • Step 1b.1 Since the initial ideas were generated by the pathologists working in Germany, the system takes German language by default. Please consider changing the prompt description in list_formatting from tasks.yaml if you desire to use other format/language as input.
  • Step 1b.2 change IDEA_DIR_PATH and IDEA_FILE_NAME to the path of the file where your own ideas are stored.
  • Step 1b.3 change LLM_FORMAT and LLM_IFORMUL to the model you want to use (similar to Idea Generation via AI
  • Step 1b.4 (Add API key if needed in main_HUMAN_PIPELINE.py). Run main_HUMAN_PIPELINE.py

• Step 2. Idea Refinement

  The idea refinement scripts utilize the ideas generated in the previous step (by default uses the ideas from ./IDEA_DATABASE/), and output the refined ideas in ./output_X/IDEA_DATABASE_REFINE/. Please change the path of the input raw ideas accordingly!)

  • Step 2.1 Open config_IREF.json: change LLM_IREF to the model you want to use (similar to Step 1 for local/API-key-based LLMs); Change the DATA_TYPE to either mplex or he for multiplex or H&E slide analysis. (Don't forget to change the IDEA_DATABASE_DIR if you have performed human-initiated ideas formulation, because the output directory for that step is ./D_IDEA_DATABASE/)

  • Step 2.2 (Add API key if needed in ./main_IREF.py). Run ./main_IREF.py

• Step 3. Idea coding

  NOTE Please make sure that you have the conda environment installed for the single cell evaluation (See installation guide).

  This code pipeline takes the refined ideas, make them into codes and store the results under "/coding_input_output/CODE_OUTPUT/idea_database_small_selection_of_ideas/

  • Step 3.1 Make sure the value for LLM_COD_REV is spark_feature_eval (Change this name accordingly if you have changed the name of the environment during its creation).
  • Step 3.2 change LLM_COD and LLM_COD_REV to the model you want to use.
  • Step 3.3 (Add API key if needed in ./main_IREF.py). Run main_COD.py.
  • Step 3.4 If you want to proceed with the single cell evaluation in the Analytical pipeline, copy both json files and python codes into a single working directory by

  mkdir ./coding_input_output/CODE_OUTPUT/merged_directory
  cp ./coding_input_output/CODE_OUTPUT/idea_database_UID_TIMESTAMP_REFINE/* ./coding_input_output/CODE_OUTPUT/merged_directory
  cp ./coding_input_output/CODE_OUTPUT/idea_database_small_selection_of_ideas/* ./coding_input_output/CODE_OUTPUT/merged_directory

  • Step 3.5 run convert_snippets_to_functions.py.

• Step 4. Parameter verification

  NOTE Be sure to run the Analytical pipeline before this step. This step is meant to verify if the results/code snippets work according to our intentions and generate meaningful results.

  This code pipeline verifies the codes and store the code snippets under ./FUNCTIONS/.

  • Step 4.1 change LLM_PV to the model you want to use (similar to Step 1 for local/API-key-based LLMs)
  • Step 4.2 run main_PARAM_VERIF.py.

  The generated/verified code snippet would look similar to this:

    import numpy as np
    import pandas as pd
    import os
    from skimage.color import rgb2hed
  
    def calculate_features_fun(Eval_Obj, sign_indices, cell_ids_in_patches, patchwise_tissue_masks, output_file_NEW): 
  
        # initialize output lists
        patch_id_list = []
        plasma_basophilia_variance = []
  
        for patch_id, cell_ids_in_patch, patch_tissue_mask in zip(sign_indices, cell_ids_in_patches, patchwise_tissue_masks):
            ...
    output_df.to_csv(output_file_NEW, index=False) 
  
    '''
    COMMENTS
    '''

For detailed explanation of the complete generative pipeline, please see Detailed documentation for generative pipeline.

<2> Analytical pipeline

Using the single cell results (e.g. from Hovernext), it is recommended to clean the geojson files based on segmentaion masks before any evaluation. The mislabelled, mis-annotated single cells can then be discarded for more accuracy. The prerequisite step is the geojson generated by single cell analysis tools such as Hovernext, segmentation map (tumor stroma and tumor are labelled).

Step 0. Preparatory steps

• Step 0.1 Go to the geojson cleaning directory.

cd ./Single_cell_analytical_pipeline/01_Hovernext_Cleaning_Preprocessing

• Step 0.2 Open ./MAIN_PostProcessing_Geojson.py and enter the path for the needed files: tumor_case_id for the tumor id (only for naming), wsi_file for the path to the WSI, mask_file for the path to the mask, geojson_file_in for the path to the raw geojson file, tissue_map (if needed) for the actual labels of the classes, and output_directory.

• Step 0.3 Run MAIN_PostProcessing_Geojson.py

Step 1a. Use case B, C and D

• Step 1a.1 Place the function (code snippet) metadata file (generated during convert_snippets_to_functions.py) under ./AI_FEATURE_FUNCTION_DIR/ and create a folder

mkdir ./AI_FEATURE_FUNCTION_DIR/function_dir/
cp /your/path/to/all/generated/and/funcionized/python/codes/ ./AI_FEATURE_FUNCTION_DIR/function_dir/

Change the path to the actual path to the functions you generated.

• Step 1a.2 go to the working directory and enter the path for the needed files: tumor_case_id for the tumor id (only for naming), wsi_file for the path to the WSI, mask_file for the path to the mask, geojson_file_in for the path to the raw geojson file, tissue_map (if needed) for the actual labels of the classes, thumbnail_file for the path to the thumnail you want to use, metadata_file for the metadata file name (only change the string part), and output_directory.

cd ./Single_cell_analytical_pipeline/02_WSI_evaluation_pipeline/

• Step 1a.3 Run ./MAIN_FUNCTION_CALLER.py.
• Step 1a.3a (Optional) if you want to use our generated functions please set the path variable function_script_directory to pathfinder_folder + "function_dir_new_mpp_scale/" in MAIN_FUNCTION_CALLER.py and use the metadata given in the ./AI_FEATURE_FUNCTION_DIR directory.

An example of single parameter output (multiple features can be available in one coded parameter):

PATCH_ID	TUMOR_COUNT	IMMUNE_COUNT	TUMOR_IMMUNE_RATIO
77	350	574	0.60975
107	1729	415	4.1662
...	...	...	...

For detailed explanation of WSI evaluation (analytical) pipeline, please see 02_WSI_Evaluation_Pipeline.

Step 1b. Tumor center and invasion front analysis

We used another analytical pipeline for the human-initiated idea generation in the publication which analyses the the WSIs with tumor center and invasion front taken into account. Of course, the users are allowed use either type of generated ideas (AI- and human-initiated) for both aforementioned use cases and the tumor center/invasion front analyses.

• Step 1a.1 Place the function (code snippet) metadata file (generated during convert_snippets_to_functions.py) under ./AI_FEATURE_FUNCTION_DIR/ and create a folder

mkdir ./AI_FEATURE_FUNCTION_DIR/function_dir/
cp /your/path/to/all/generated/and/funcionized/python/codes/ ./AI_FEATURE_FUNCTION_DIR/function_dir/

Change the path to the actual path to the functions you generated.

• Step 1b.2 go to the working directory and open file_loader.py. enter the path for the needed files: WSI_folder for the path to the WSI, mask_folder for the path to the mask (only tumor and tumor stroma labelled), geojson_folder for the path to the raw geojson file, mask_raw_folder for the path to multi-class segmentation masks, metadata_file for the metadata file name (only change the string part), and output_directory.

cd ./03_Tumor_Center_Invasion_Front/

• Step 1a.3 Run ./MAIN_FUNCTION_CALLER.py.
• Step 1a.2a (Optional) if you want to use our generated functions please set the path variable function_script_directory to pathfinder_folder + "function_dir_tumor_center_invasion_front/" in MAIN_FUNCTION_CALLER.py and use the metadata file human_function_metadata_496_suc.json given in the ./AI_FEATURE_FUNCTION_DIR directory.

The output looks similar to that from other use cases. The only difference is the added inner/outer labels

PATCH_ID	INNER/OUTER	LYMPHOCYTE_COUNT	NEUTROPHIL_COUNT	LYMPHOCYTE_TO_NEUTROPHIL_RATIO
375	Inner	43	2	21.5
376	Inner	77	1	77.0
...	...	...	...	...

For detailed explanation of the tumor center and invasion frount analysis, please see 03_Tumor_Center_Invasion_Front.

<3> Prognostic Pipeline

This Pipeline is the application of the features generated from the coded ideas in patient survival analyses.

Step 1. Configure the pipeline

(be sure that you are in the working directory ./Generative_and_Prognostic_Pipeline)

• Step 1.1 Open config_PROGN_EVAL.json and change CSV_MAIN_DIR to the path to the results after analytical pipeline, DATABASE_FILE to the path to the clinical data (By default the clinical data contains pN, pN_KOR,pT_KOR, pT, CSS_EVENT, PFS_EVENT, STAGE, GENDER, AGE, OS_EVENT, ER, PR, HER2NEU_IHC_FISH, HISTOTYPE. Please see the detailed explanation of eahc of these entry here. If your csv does not contain some columns, please consider remove the items in the list variables in PARAMETERS_CORRELATION from the config file and CLINICAL_VARIABLES and CLINICAL_COVARIATES from main_PROGN_EVAL_P2_ANALYSIS.py later on.)
• Step 1.2 Run main_PROGN_EVAL_P1_PREPARE_DATABASE.py and main_PROGN_EVAL_P2_ANALYSIS.py sequentially.

<4> Additional Analysis

If you wish to perform additional analyses just like in publication, please follow the guidelines below. This section contains the tutorial for how to analyse semantic distances between ideas, and the Person's correlation test between semantic distance and similarity scores.

1. Semantic distance calculation: Firstly install the environment and activate it. Then place the final idea databases you want to analyse into one single folder (it also supports multiple idea database. The idea database files are generated, by default, in .../output_X/IDEA_DATABASE/idea_database_* during idea generation)

conda env create -f spark_stats.yaml
conda activate spark_stats
cd Statistical_Analysis
python Semantic_Distance.py -n /your/path/to/the/idea/database

This will produce a single image showing the embedding points projected into a 2D principal-component space, similar to the example below.

2. Correlation between semantic distance and similarity scores: Similar to the previous step, please make sure to place both idea_database_ and duplicate_idea_database_* files in the same path and run

conda env create -f spark_stats.yaml
conda activate spark_stats
cd Statistical_Analysis
python Correlation_Similarity_Semantic_Distance.py -n /your/path/to/the/idea/database

This will print output like GPT-OSS-120B: Pearson correlation: 0.517 directly to the command prompt.

More Setup

An overview of the output directory structure and the adjustable parameters for the generative pipeline can be found in More Setup for SPARK

Issues

• Please report all issues on the public forum.

License

© Tolkach Lab - This code is made available under Attribution-NonCommercial-ShareAlike 4.0 International license.

Funding

Reference

If you find our work useful in your research or if you use parts of this code please consider citing our paper:

@article{Trost2026,
  title = {An agentic framework for autonomous scientific discovery in cancer pathology},
  ISSN = {1546-170X},
  url = {http://dx.doi.org/10.1038/s41591-026-04357-y},
  DOI = {10.1038/s41591-026-04357-y},
  journal = {Nature Medicine},
  publisher = {Springer Science and Business Media LLC},
  author = {Trost,  Florian and Zhang,  Bide and Aring,  Ines and Bauer,  Marcus and Glamann,  Lennert and Wessolly,  Michael and Johnson,  Kyra and G\"{o}bel,  Heike and Lerbs,  Tristan and Sangenne,  Taban and Herrmann,  Peter and Mairinger,  Fabian and Kopp,  Christopher and Michels,  Sebastian and Rasokat,  Anna and Heldwein,  Matthias and Wagner,  Steffen and Sch\"{o}mig-Markiefka,  Birgid and Wolf,  J\"{u}rgen and Hartmann,  Sylvia and Wickenhauser,  Claudia and Bychkov,  Andrey and Klussmann,  Jens Peter and Quaas,  Alexander and Buettner,  Reinhard and Tolkach,  Yuri},
  year = {2026},
  month = Apr 
}