ModelTrace is a machine learning system that identifies the source language model of any given text response. Using a 4-model weighted voting ensemble with SHAP-based explainability, it achieves 83.28% accuracy in distinguishing between five state-of-the-art language models.
Quick Question It Answers:
"Which language model wrote this response?"
- GPT-4o (OpenAI) - Formal, structured responses
- DeepSeek V3 (DeepSeek) - Technical, analytical approach
- Gemma 4 (Google) - Verbose, detailed explanations
- Llama 3.3 (Meta) - Balanced, consistent style
- Qwen 2.5 (Alibaba) - Concise, direct communication
- 83.28% overall accuracy on diverse prompts
- 98% recall for Gemma (highly distinctive)
- 95% recall for DeepSeek (technical markers)
- 87% recall for Llama (balanced style)
- SHAP-based feature contributions showing why predictions are made
- Top 3 contributing features for each prediction
- Feature importance rankings for global understanding
- Confidence scores indicating prediction certainty
- 4-model weighted ensemble for robustness
- <120ms inference latency per prediction
- REST API ready for deployment
- Model serialization for easy deployment
- 85 features from text analysis
- 22 stylometric (structure, readability)
- 13 linguistic (tone, communication style)
- 50 semantic (deep meaning via embeddings)
- PCA dimensionality reduction from 384→50 dimensions
- Cross-validated performance across all splits
| Metric | Value |
|---|---|
| Accuracy | 83.28% |
| Test Samples | 300 |
| Models Trained | 5 baseline + ensemble |
| Features Used | 85 dimensions |
| Inference Latency | 80-120ms |
| Throughput | 8-12 predictions/sec |
| Model | Precision | Recall | F1-Score | Status |
|---|---|---|---|---|
| DeepSeek | 0.93 | 0.95 | 0.94 | Excellent |
| Gemma | 0.95 | 0.98 | 0.97 | Excellent |
| Llama | 0.88 | 0.87 | 0.87 | Good |
| GPT-4o | 0.71 | 0.70 | 0.71 | Moderate |
| Qwen | 0.69 | 0.68 | 0.69 | Moderate |
Predicted
Actual DS GM G4 LL QW
────────────────────────────────
DeepSeek 57 0 2 0 1
Gemma 0 58 0 0 1
GPT-4o 3 2 42 1 12
Llama 1 0 3 52 4
Qwen 0 1 12 6 41
# Clone repository
git clone https://github.com/yourusername/ModelTrace.git
cd ModelTrace
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txtfrom modelrace_inference import predict_and_explain_production
# Make prediction
text = "Your model response here..."
prediction, confidence_scores, top_features = predict_and_explain_production(text)
# Output
print(f"Predicted Model: {prediction}")
print(f"Confidence: {confidence_scores[prediction]:.1%}")
print("\nTop Contributing Features:")
for feature, contribution in top_features:
print(f" - {feature}: {contribution:.3f}")streamlit run streamlit_app.pyOpens interactive web interface at http://localhost:8501
Raw Text
↓
Feature Extraction (85 features)
├─ Stylometric (22): Length, readability, punctuation
├─ Linguistic (13): Tone, formality, reasoning markers
└─ Semantic (50): Embeddings via sentence-transformers
↓
Feature Scaling (StandardScaler)
↓
PCA Reduction (384 → 50 dimensions)
↓
4-Model Weighted Ensemble
├─ Gradient Boosting (weight 2.5)
├─ Logistic Regression (weight 2.5)
├─ LightGBM (weight 1.5) → SHAP extraction
└─ Random Forest (weight 1.0)
↓
Weighted Voting
↓
Prediction + Confidence + SHAP Explanations
| Model | Weight | Baseline | Role |
|---|---|---|---|
| Gradient Boosting | 2.5 | 82% | Primary predictor |
| Logistic Regression | 2.5 | 81% | Interpretability |
| LightGBM | 1.5 | 78% | SHAP explanations |
| Random Forest | 1.0 | 76% | Diversity |
Stylometric Features (22):
- Response length, sentence structure
- Flesch Reading Ease, Gunning Fog Index
- Punctuation frequencies
- Vocabulary diversity
Linguistic Features (13):
- Hedging and confidence words
- Formal language markers
- Pronoun patterns
- Reasoning markers
Semantic Features (50):
- Embeddings from
sentence-transformers(all-MiniLM-L6-v2) - PCA reduced from 384 to 50 dimensions
- Captures deep semantic meaning
Global Level:
# Feature importance for overall decision making
importances = lgb_submodel.feature_importances_Local Level (SHAP):
# Why this specific prediction was made
contributions = lgb_submodel.booster_.predict(
scaled_features,
pred_contrib=True
)from src.inference import predict_and_explain_production
text = """
The algorithm works by first collecting raw data, then processing it through
multiple stages. Each stage applies transformations that gradually extract
meaningful patterns. The final output represents learned features in
lower-dimensional space.
"""
model, scores, features = predict_and_explain_production(text)
# Output:
# Predicted Model: DeepSeek
# Confidence: 89%
# Top Features: ['Semantic Embedding 2', 'Reasoning Markers', 'Sentence Length']import pandas as pd
from src.inference import batch_predict
# Load responses
df = pd.read_csv('responses.csv')
# Get predictions
results = batch_predict(df['response'].tolist())
# Create results dataframe
results_df = pd.DataFrame(results)
results_df.to_csv('predictions.csv', index=False)# Start API server
python -m uvicorn api.main:app --reload
# Make prediction request
curl -X POST "http://localhost:8000/predict" \
-H "Content-Type: application/json" \
-d '{"text": "Your response text here"}'
# Response:
# {
# "prediction": "GPT-4o",
# "confidence": 0.87,
# "confidence_scores": {...},
# "top_features": [...]
# }Complete documentation available in the Documentation/ folder:
-
PROJECT_DOCUMENTATION.md - Comprehensive project guide covering:
- System architecture
- Feature engineering details
- Model training approach
- Ensemble configuration
- Technical implementation
-
ANALYSIS_DOCUMENTATION.md - Results and findings including:
- Individual model performance
- Ensemble evaluation
- Confusion matrix analysis
- Per-model metrics
- Error analysis
-
GPT4o_vs_Qwen_Analysis.md - Deep dive into the most challenging confusion pair