SIH 2023 — R&D Performance Dashboard

End-to-end analytics pipeline aligned with the Klüber R&D Dashboard JD:
"Development of a digital Performance Dashboard for R&D and Innovation Portfolio Management"

---

What this project does

Stage	Description
Scrape	Config-driven scraper with retry logic, rate limiting, raw data lineage
Clean	Fixes 9 original flaws (leakage, batch extraction, status normalisation, duplicates)
SQL Database	SQLite with WAL mode — 4 tables, indexes, KPI storage
KPI Engine	23 KPIs across 5 groups: pipeline, geographic, category, organisation, data quality
ML Model	Fixed sklearn Pipeline — no leakage, scaling inside split, Stratified K-Fold CV
Dashboard	Premium Plotly Dash — 4 pages, dark OLED theme, SVG icons, 150–300ms transitions

---

Quick Start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Run full pipeline (clean → DB → KPIs → ML)
python pipeline.py --steps clean db kpi ml

# 3. Launch dashboard
python dashboard/dash_app/app.py
# Open http://127.0.0.1:8050

---

Project Structure

SIH_scrape_analysis_prediction/
├── pipeline.py                   # Orchestrator — run everything
├── config/
│   └── config.yaml               # All settings (URLs, DB path, ML params)
├── src/
│   ├── ingestion/
│   │   └── scraper.py            # Layer 1: Config-driven scraper
│   ├── processing/
│   │   └── clean.py              # Layer 2: Clean + fix all 9 flaws
│   ├── database/
│   │   └── db.py                 # Layer 3: SQLite manager
│   ├── kpi/
│   │   └── engine.py             # Layer 4: 23 KPIs → JSON/CSV/DB
│   └── ml/
│       └── model.py              # Layer 5: Fixed ML pipeline
├── dashboard/
│   └── dash_app/
│       ├── app.py                # Plotly Dash 4-page dashboard
│       └── assets/style.css      # UI/UX Pro Max design system
├── data/
│   ├── raw/                      # Timestamped raw scrape files
│   ├── processed/                # Cleaned DataFrames (timestamped)
│   ├── exports/                  # KPI reports (JSON + CSV)
│   └── models/                   # Trained ML models (.pkl)
├── logs/                         # Pipeline run logs
├── New/                          # Source CSVs (original data)
└── requirements.txt

---

Dashboard Pages

Page	Charts
Overview	8 KPI cards, Selection Funnel, Category Donut, Screening Status, State Bar
Screening	3 live filters → Org Bar + State Bar + Data Table
Finals & Winners	Award Breakdown, Top Colleges, Nodal Centres, Prize Treemap, Winners Table
KPIs	All 23 KPIs with formulas, KPI Definition Table

Design System (ui-ux-pro-max skill):

• OLED dark palette (#080C14 / #0F172A / #3B82F6)
• Fonts: Inter (labels) + Fira Code (numbers)
• Icons: DashIconify Heroicons — no emojis
• Animations: 150–300ms ease-out, prefers-reduced-motion respected
• Treemap: white 2–3px borders (skill chart rule)
• Funnel: gradient stages with conversion % labels

---

Original Flaws Fixed

#	Flaw	Fix
1	Label leakage (PSID in features)	PSID / TEAM ID / TEAM NAME removed before split
2	Wrong batch extraction	Regex on PS ID digits → Batch 15/16/17
3	Unscaled test data	StandardScaler inside sklearn Pipeline
4	Missing value handling	Per-column null threshold with logging
5	Duplicate rows	drop_duplicates() with before/after count
6	Wrong batch column	Fixed via PS ID regex
7	Status case inconsistency	.str.upper().str.strip() on all status cols
8	Column whitespace	.str.strip() on all column names at load
9	Combined CSV lineage	Timestamped processed files in data/processed/

---

Pipeline Usage

# All steps
python pipeline.py

# Selective steps
python pipeline.py --steps clean db kpi      # skip ingest, skip ml
python pipeline.py --steps ml                # retrain model only

# Individual modules
python src/processing/clean.py               # cleaning only
python src/database/db.py                    # DB init + populate
python src/kpi/engine.py                     # compute + export KPIs
python src/ml/model.py                       # train + evaluate model

---

KPIs Computed

Group	KPIs
Pipeline	total_screened, total_finalists, total_winners, selection_rate, win_rate, prize_pool_inr, avg_prize_winner
Geographic	unique_states, top_state, top_state_count, state_hhi
Category	software_teams, hardware_teams, software_pct, hardware_pct
Organisation	unique_orgs, top_org, top_org_teams, avg_teams_per_org
Data Quality	data_completeness_pct, null_cell_count, duplicate_screening, duplicate_finals

---

Requirements

dash>=4.0
dash-bootstrap-components
dash-iconify
plotly
pandas
sqlalchemy
scikit-learn
pyyaml
requests

Install: pip install -r requirements.txt