Replication Package — Uncovering the Cognitive Biases of LLMs in Software Effort Estimation

This package contains all materials required to replicate the experiments and reproduce the results reported in the dissertation. It accompanies the dissertation titled:

Uncovering the Cognitive Biases of LLMs in Software Effort Estimation

Repository Structure

├── experiment_selection/       # Systematic review process and paper selection
├── experiments/                # CLI-driven Python packages implementing all 16 experiments
│   ├── main.py                 # Single entry point: run one paper's experiment(s)
│   ├── utils.py                # Shared utility code for all packages
│   └── ...
├── figures/                    # Figures appearing in the dissertation and the script to generate them
├── requirements_documents/     # All requirements documents used as estimation inputs
├── statistical_analysis/       # R script and output for all statistical tests
└── results/                    # Numerical outputs from all experiments

Prerequisites

Python

Install dependencies with:

pip install openai pandas

An OpenRouter API key is required to run the experiments. Set it as an environment variable before running any experiment:

export OPENROUTER_API_KEY=your_key_here

R

R can be obtained from https://www.r-project.org/. The following packages are required:

install.packages("jsonlite")
install.packages("effsize")

TAWOS

The requirements documents were constructed from the TAWOS dataset, which must be obtained and set up independently if you wish to regenerate them from source. The dataset is publicly available at:

https://doi.org/10.1145/3524842.3528029

The SQL queries used to extract source data are provided in requirements_documents/issues.sql and requirements_documents/summary.sql. Pre-constructed documents are provided in the requirements_documents/ folder and do not need to be regenerated to run the experiments.

Replication Steps

Follow these steps in order to fully replicate the study.

1. Run the Experiments

Each paper is implemented as a Python package under experiments/, run via the single CLI entry point experiments/main.py. Run it from the repository root (not from inside experiments/) so that the relative paths to requirements_documents/ and results/ resolve correctly:

python experiments/main.py <paper> [<study>]

Run with no arguments to see full usage, including every paper and the studies available for it. Omitting <study> runs every study for that paper in order and writes the same results file described below.

Paper	Source Paper	Bias Category
aranda2005	Aranda & Easterbrook (2005)	Anchoring
lohre2014	Løhre & Jørgensen (2014)	Anchoring
haugen2006	Haugen (2006)	Over-optimism
molokken2003	Moløkken & Jørgensen (2003)	Over-optimism
jorgensen2009	Jørgensen (2009)	Over-optimism
connolly1997	Connolly & Dean (1997)	Over-confidence
jorgensen2002	Jørgensen et al. (2002)	Over-confidence

For example:

python experiments/main.py jorgensen2009            # run all 4 experiments, write combined results
python experiments/main.py jorgensen2009 experiment_a  # run just Experiment A

See experiments/README.md for the full command reference and package layout.

Note that regenerated results may differ slightly from those provided due to implementation-level non-determinism in model providers, even at temperature zero. See Section 3.2 of the dissertation for a discussion of this limitation.

2. Run the Statistical Analysis

With the results/ folder populated, run statistical_analysis/analysis.R from the same directory as results/. This will produce output.md containing all test statistics, corrected p-values, and effect sizes.

3. Regenerate Figures

With the results/ folder populated, run figures/visualise_results.py from the same directory as results/. This will regenerate all figures reported in the dissertation.

Notes

• Full methodological details, including prompt structure, experimental design, and deviations from original study designs, are documented in the dissertation.
• Failure logs were used during execution to guide post-hoc recovery of malformed responses but were not retained as artefacts. Recovery logic is implemented within utils.py.
• Each subdirectory contains its own README with further details.