LlamaMTS: Llama Metastasis Instruction Tuning with LoRA

This is the code repository for LlamaMTS: Optimizing Metastasis Detection with Llama Instruction Tuning and BERT-Based Ensemble in Italian Clinical Reports (Lilli et al., 2024), published at NAACL ClinicalNLP 2024.

In this paper, we introduce LlamaMTS, a fine-tuned Llama model adapted through the LoRA instruction tuning technique. The model is designed to identify the presence of tumoral metastasis by analyzing Electronic Health Records (EHRs) from patients diagnosed with breast cancer. LlamaMTS uses Camoscio (Santilli and Rodola, 2023) as the base Italian Llama adaptation; this repository also follows the Camoscio codebase approach for LoRA adaptation. To further improve performance, we use an ensemble strategy that incorporates a BERT-based model fine-tuned on the same classification task.

Because full EHRs may exceed the maximum token length supported by Llama during training, the pipeline also includes a summarization step over training and testing data. This produces shorter, more focused texts, preserving information about metastasis, lesions, nodules, metabolic activity, and staging while reducing noise from long clinical reports.

LLamaMTS Paper: https://aclanthology.org/2024.clinicalnlp-1.13/
Camoscio Paper: https://aclanthology.org/2023.clicit-1.46/

Repository Structure

llm-healthcare/
├── config/
│   ├── bert.yaml          # BERT fine-tuning and evaluation parameters
│   ├── ensemble.yaml      # Ensemble strategy, model inputs, weights, and output paths
│   ├── llama_mts.yaml     # LlamaMTS fine-tuning and evaluation parameters
│   └── llm.yaml           # LLM summarization and zero-shot benchmark parameters
├── ensemble/
│   └── ensemble.py        # Unified max/average ensemble procedure
├── evaluation/
│   ├── bert/evaluation.py
│   ├── llama_mts/evaluate.py
│   └── llm/classify_data.py
├── src/
│   ├── bert/finetune.py
│   ├── llama_mts/scripts/finetune.py
│   └── llm/summarize_data.py
└── requirements.txt

Installation

git clone https://github.com/LivLilli/llm-healthcare.git
cd llm-healthcare

python -m venv venv
source venv/bin/activate

pip install -r requirements.txt

Some procedures require GPU support, Hugging Face model access, and local model/checkpoint paths configured in the YAML files. The zero-shot LLM benchmark and summarization scripts use Ollama, so the selected Ollama models must be available locally.

Configuration

All configurable parameters are stored in config/.

config/llm.yaml controls:

• summarization preprocessing with src/llm/summarize_data.py
• zero-shot LLM classification benchmark with evaluation/llm/classify_data.py
• model name, prompt, input file, text column, output folders, output column, and save frequency

config/llama_mts.yaml controls:

• LoRA fine-tuning of LlamaMTS with src/llama_mts/scripts/finetune.py
• LlamaMTS inference with evaluation/llama_mts/evaluate.py
• base model, tokenizer, checkpoint paths, training hyperparameters, LoRA settings, generation settings, and output folders

config/bert.yaml controls:

• BERT fine-tuning with src/bert/finetune.py
• BERT inference with evaluation/bert/evaluation.py
• model name, dataset path, checkpoint, labels, training arguments, and output folders

config/ensemble.yaml controls:

• ensemble inference with ensemble/ensemble.py
• strategy selection: max or average
• average weighting metric: auc, f1, or weight
• prediction input files, column normalization, decision threshold, and output folders

Pipeline

1. Summarization Preprocessing

Use LLM-based summarization to create concise clinical text before model training and testing:

python src/llm/summarize_data.py

Edit config/llm.yaml under summarize_data to set the Ollama model, prompt, input file, text column, and output folder.

2. LlamaMTS Adaptation

Fine-tune the Camoscio-based Llama model with LoRA instruction tuning to produce LlamaMTS:

python src/llama_mts/scripts/finetune.py

Parameters for the base model, tokenizer, dataset, LoRA configuration, training hyperparameters, and output checkpoint folder are in config/llama_mts.yaml under finetune.

3. BERT-Based Model Fine-Tuning

Fine-tune the BERT-based classifier on the same metastasis classification task:

python src/bert/finetune.py

Parameters are in config/bert.yaml under finetune.

4. Inference

Run LlamaMTS inference:

python evaluation/llama_mts/evaluate.py

Run BERT inference:

python evaluation/bert/evaluation.py

The corresponding evaluation sections in config/llama_mts.yaml and config/bert.yaml define checkpoint paths, test data, experiment names, and output folders.

5. Ensemble

Combine the LlamaMTS and BERT predictions:

python ensemble/ensemble.py

Set the ensemble strategy in config/ensemble.yaml:

ensemble:
  strategy: "average"          # "average" or "max"
  average_weight_metric: "auc" # "auc", "f1", or "weight"

The average ensemble computes class probabilities using model-level weights. The max ensemble selects, for each document, the prediction with the highest confidence.

6. Zero-Shot Benchmark

For benchmark analysis, the repository includes zero-shot prompting with other local LLMs through Ollama:

python evaluation/llm/classify_data.py

This procedure is configured in config/llm.yaml under classify_data and can be used to compare LlamaMTS and BERT-based models against prompted LLM baselines.

Citation

If you use this code, please cite LlamaMTS:

@inproceedings{lilli-etal-2024-llamamts,
    title = "{L}lama{MTS}: Optimizing Metastasis Detection with Llama Instruction Tuning and {BERT}-Based Ensemble in {I}talian Clinical Reports",
    author = "Lilli, Livia and
      Patarnello, Stefano and
      Masciocchi, Carlotta and
      Masiello, Valeria and
      Marazzi, Fabio and
      Tagliaferri, Luca and
      Capocchiano, Nikola Dino",
    booktitle = "Proceedings of the 6th Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.clinicalnlp-1.13/",
    doi = "10.18653/v1/2024.clinicalnlp-1.13",
    pages = "162--171"
}

Please also cite Camoscio when referring to the Italian Llama adaptation:

@inproceedings{santilli-rodola-2023-camoscio,
    title = "{C}amoscio: An {I}talian Instruction-tuned {LL}a{MA}",
    author = "Santilli, Andrea and
      Rodol{\`a}, Emanuele",
    booktitle = "Proceedings of the Ninth Italian Conference on Computational Linguistics (CLiC-it 2023)",
    year = "2023",
    address = "Venice, Italy",
    publisher = "CEUR Workshop Proceedings",
    url = "https://aclanthology.org/2023.clicit-1.46/",
    pages = "385--395"
}