Thyroid Cancer Recurrence: ML & Statistical Analysis

This project provides a comprehensive data science workflow to analyze and predict thyroid cancer recurrence based on patient clinical data. By combining rigorous statistical testing with state-of-the-art machine learning algorithms, the goal is to identify key risk factors and build a high-accuracy predictive model.

📂 Project Structure

• scripts/: Modular Python scripts for specific tasks.

  • dataprocess.py: Handles data cleaning, Min-Max normalization for Age, and Label Encoding for categorical features.
  • normaldistribution.py: Performs distribution analysis and normality testing (Shapiro-Wilk & Kolmogorov-Smirnov).
  • statisticalcorrelation.py: Analyzes dependencies between categorical variables using Chi-square ($X^2$) tests and heatmaps.
  • agecorrelation.py: Investigates Age variance across different clinical categories using the Kruskal-Wallis test.
  • crossvalidation.py: Trains and compares classification models (XGBoost, Random Forest, SVM, Logistic Regression) using Stratified K-Fold Cross-Validation.

• notebook.ipynb: An interactive Jupyter Notebook documenting the entire end-to-end analysis.

Key Methodology

1. Statistical Validation

Before modeling, we perform extensive Exploratory Data Analysis (EDA). We use:

• Chi-square ($X^2$) Test: To identify significant correlations between categorical variables like 'Stage', 'Risk', and 'Recurrence'. Only results with $p < 0.001$ are visualized in the correlation heatmap.
• Kruskal-Wallis Test: A non-parametric method used to determine if the distribution of 'Age' differs significantly across clinical subcategories.

2. Machine Learning Pipeline

We evaluate multiple classifiers to predict the 'Recurred' variable:

• Evaluation Strategy: 5-Fold Stratified Cross-Validation to ensure model stability and generalizability.
• Metrics: Models are assessed based on Accuracy, Precision, Recall, F1-Score, and ROC-AUC.
• Convex Hull Visualization: Used on ROC curves to identify the optimal performance frontier.

Results

• Top Models: XGBoost and Random Forest achieved the best results, with Mean Accuracy and AUC scores exceeding 95%.
• Insights: Clinical features such as 'Pathology', 'T' (Tumor size), and 'Risk' were found to have the strongest statistical associations with cancer recurrence.

Installation & Usage

1. Clone the repository:

git clone https://github.com/lampriana15/ThyroidCancerRecurrence-ML-StatisticalAnalysis.git

2. Install dependencies:

pip install -r requirements.txt

3. Run the analysis:

• Execute python scripts/dataprocess.py first to generate the processed data.
• Run any other script in the scripts/ folder or open the notebook.

📜 License & Data Source

• Dataset: Differentiated Thyroid Cancer Recurrence from Kaggle by Rahul Kumar.
• License: CC BY-NC-SA 4.0.