This file containes the detailed information about each sub-modules in the generative pipeline. Examples of how the output looks like and their format are also attached for clarity.
1.0. Multi-agentic backbone This work utilizes CREWAI as the multi-agentic AI backbone. Within this framework, multiple specialized agents collaborate to complete complex tasks. Each agent is assigned a specific role, objective and skill set (backstory). In our setup, multiple collaborating agents form a crew, which acts like a project team involving specialists based on the role they are assigned.
1.1. Idea generation module
This module is consisted of three major agents:
1.1.1a Idea generation sub-module:
Users are free to define the number of cycles and number of generated ideas in each cycle for the idea generation module. There are possibilities to configure the level of creativity, number of cell types for the idea generation (e.g. ask the model to generate 10 creative ideas for 4 distinct cell typles. Repeat this process for 4 cycles). More information about how to configure can be obtained in the section xxx. This part generates the ideas in the following format:
Okay, I need to generate 10 ideas for analyzing single-cell and tissue data to predict patient survival or tumor progression. The data includes H&E-stained images with tissue...
</think>
```json
[
{
"idea_number": 1,
"idea_title": "Tumor Cell Density in Epithelial Compartment",
"cancer_type": "multiple cancer types",
"cell_type_code": "T",
"biological_hypothesis": "Higher density of tumor cells in the epithelial compartment is associated with worse patient survival.",
"analytical_methodological_implementation": "Count the number of tumor cells per unit area in the epithelial compartment using the GeoJSON files.",
"estimated_complexity": 2,
"parameter_type": "cell counting",
"analysis_principle": "direct measurement"
},
{
"idea_number": 2,
...
}]1.1.1b Human-initiated ideas: Not only does SPARK allows AI-generated ideas from scratch, but also it provides an alternative sub-module to use outlined ideas thought/given by the users. Human analysts can provide their ideas in minimally structured free-text form (e.g., “1. Number of lymphocytes close to tumor cells. 2. Density of macrophages in tumor stroma.”). The human-initiated ideas (please see above image) are first formatted by the list-formatting agent, and formulated in a similar structure of ideas in SPARK-initiated idea generation pipeline. This enables the ideas to be refined and coded later on by the same downstream sub-modules during generative pipeline.
1.1.2 Idea review agent:
The idea reviewing agent ensures each idea adheres strictly to the data modality (only use the available input data), invovles the predefined number of different cell types (for Use case 1) etc. The idea would be labelled as "rejected" if it violates the guideline principles defined during prompting. The reviewed ideas are put into the format that can be processed in downstream steps:
[
{
"idea_number": 1,
"idea_title": "Tumor Cell Density in Epithelial Compartment",
...
"rejection": "no",
"rejection_reason": ""
},
...
{
{
"idea_number": 4,
"idea_title": "Macrophage Orientation Relative to Tumor Cells",
"cancer_type": "multiple cancer types",
"cell_type_code": "M",
...
"rejection": "yes",
"rejection_reason": "Involves two distinct cell types (macrophages and tumor cells) instead of exactly one as required."
},
...
}]1.1.3 Duplicate detection agent:
As the cycle of generation proceeds, ideas that pass the reviewing process would be stored in the idea database. The newly generated ideas are checked for duplication in comparison with each single idea available in the database. The duplicate agent scales the level of similarity between the two selected ideas from 1 to 10. Users are free to select a threshold (8 for the use cases in our publication) for the similarity score. Any ideas above that threshold would be considered as a duplicate, and prevented from being added to the idea database. To trace back the similarity scores, the scoring of each investigated idea is also documented in the json file in ./output_X/IDEA_DATABASE/duplicate_xx.json:
[
{
"idea_title": "Neutrophil Count in Tumor Stroma",
"cancer_type": "multiple cancer types",
"cell_type_code": "N",
"biological_hypothesis": "Higher neutrophil count in the tumor stroma is associated with aggressive tumor behavior and worse prognosis.",
"analytical_methodological_implementation": "Count neutrophils within the stromal compartment using the GeoJSON files.",
"estimated_complexity": 2,
"parameter_type": "cell counting",
"analysis_principle": "direct measurement",
"similarity_scores": [
4,
2,
9, # considered as duplciate in the use case 1, 2, and 3.
4
],
"most_similar_idea": 3,
"METADATA_FILE_NAME": "metadata_hjapyx_20251018_194336.json",
"idea_number": 1,
"idea_uid": "cfrp1yaxogtz"
},
{
...
}]1.2 Idea refinement sub-module
During this step, the Idea Refinement Agent (IREFA) takes one idea at a time from the Raw Idea Database and, using the given task description (including information on available data, coding structure of the WSI pipeline, and the structure and coding nomenclature of WSI object features - i.e., EvalObj, described in the next section on preprocessing), provides a detailed, step-by-step analytical and technical implementation of the parameter(s) defined in the idea. In other words, it specifies a clearly defined process from the WSI object to the parameter outputs for that WSI. All refined ideas, now containing the added technical implementation, are saved in the Refined Idea Database (JSON file) to be sampled for coding in the next step.
[
{
"idea_title": "Tumor Cell Nuclear Area Variation",
"cancer_type": "multiple cancer types",
"cell_type_code": "T",
"biological_hypothesis": "Larger nuclear areas of tumor cells reflect higher proliferation and correlate with more aggressive disease course.",
"analytical_methodological_implementation": "Compute the polygon area for each tumor cell nucleus and calculate the mean and variance across the entire slide.",
"estimated_complexity": 2,
"parameter_type": "other",
"analysis_principle": "direct measurement",
"idea_quality": "basic",
"cycle": 1,
"global_cycle_counter": 1,
"METADATA_FILE_NAME": "metadata_1bi9gh_20250209_135855.json",
"idea_number": 1,
"idea_uid": "whgod6yxdsrv",
"decision_refinement": "feasible",
"decision_refinement_clarification": "",
"implementation_steps": {
"implementation_step_1": "Filter the list of cell objects by selecting only those with cell_class code 'T' (tumor cells) and that reside in the 'Tumor' compartment as defined by the in_mask_set property.",
"implementation_step_2": "For each filtered tumor cell, access the polygon property and compute its area using the polygon.area method provided by Shapely.",
...
"implementation_step_7": "Integrate this code snippet within the existing Python script that already loads and preprocesses the cell objects, ensuring that the final CSV file is generated at the end of the whole-slide (or patch level) analysis."
}
},
{
...
}]1.3 Idea coding sub-module
1.3.1 Idea coding agent Next, the refined ideas, with specific step-wise implementation guidance, the coding agent converts the ideas into actual code snippets. The coding agent first attempts to develop python codes for the idea according to the given format and guidance. An examplified json output is as follows:
```json
{
"idea_uid": "0fuzp53opov4",
"tissue_types": "ALL",
"cell_type_code": "P",
"patch_size_mode": "standard_1000m",
"python_code": "import numpy as np\\nimport pandas as pd\\nimport os\\nfrom skimage.color import rgb2hed\\n\\n# initialize output lists\\npatch_id_list = []\\nplasma_basophilia_variance = []\\n\\nfor patch_id, cell_ids_in_patch, patch_tissue_mask in zip(sign_indices, cell_ids_in_patches, patchwise_tissue_masks):\\n ... cells_objects_in_patch = [Eval_Obj.single_cell_list[i] for i in cell_ids_in_patch]\\n for cell_object in cells_objects_in_patch:\\n # filter plasma cells\\n if cell_object.cell_class != 'Plasma':\\n continue\\n ... output_df.to_csv(output_file, index=False)\\n'''COMMENTS\\n'''"
}```
1.3.2 Idea review agent
Once the code is generated, the snippet is then used as a function in a subprocess for a fast-load evaluation of single cell data. If the code is executed with error-free outcome, the coded idea is stored in the output directory. In the case if it fails to generate analysable code, the review agent tries to correct the code that resolve the issues. Notably, if the runtime for the code snippet on the fast-load case is slower than 60 seconds, it is also considered as a failure. An example is shown below:
import numpy as np
import pandas as pd
import os
from skimage.color import rgb2hed
# initialize output lists
patch_id_list = []
plasma_basophilia_variance = []
for patch_id, cell_ids_in_patch, patch_tissue_mask in zip(sign_indices, cell_ids_in_patches, patchwise_tissue_masks):
# collect intensities for plasma cells in this patch
intensities = []
# create list of cell objects in the current patch
cells_objects_in_patch = [Eval_Obj.single_cell_list[i] for i in cell_ids_in_patch]
for cell_object in cells_objects_in_patch:
# filter plasma cells
if cell_object.cell_class != 'Plasma':
continue
...
output_df.to_csv(output_file, index=False)
assert os.path.exists(output_file), 'NO CSV OUTPUT GENERATED'
'''
COMMENTS
'''1.3.3 Conversion of code snippets into callable functions
An external python script, used for convertion of the snippets into callable functions, ensures the unification and standardization of the function name, input of arguments, output forma. Additionally, it extracts and cleans metadata from associated JSON files (removing embedded code), updates them to reference the new function scripts, and saves all metadata into a consolidated JSON file for further use.
import numpy as np
import pandas as pd
import os
from skimage.color import rgb2hed
def calculate_features_fun(Eval_Obj, sign_indices, cell_ids_in_patches, patchwise_tissue_masks, output_file_NEW):
# initialize output lists
patch_id_list = []
plasma_basophilia_variance = []
for patch_id, cell_ids_in_patch, patch_tissue_mask in zip(sign_indices, cell_ids_in_patches, patchwise_tissue_masks):
...
output_df.to_csv(output_file_NEW, index=False)
'''
COMMENTS
'''1.4 Parameter verification sub-module
After successful coding, the parameters are implemented across the entire cohort using our WSI analysis pipeline. Each snippet is converted into a callable function, generating one CSV file per WSI and idea, containing regional measurements (patches = rows) for all coded parameters. A subsequent agentic module processes these CSVs to verify parameters before downstream clinical analysis. This review combines non-agentic and agentic components: the non-agentic part excludes parameters with >80% zero or NA values, while the agentic part reviews the idea description, code snippet, and parameter comments to exclude non-normalized (e.g., raw counts), non-numeric, or inconsistent parameters. The outcome is two lists — excluded and verified parameters — with the latter used for downstream analyses.