BoardGameGeek End-to-End Recommender Pipeline

This repository hosts the complete lifecycle of a BoardGameGeek recommendation engine, spanning from web scraping and data refinement to the implementation of advanced machine learning models. The project is structured as a modular pipeline, allowing users to enter at any stage: from raw data acquisition to hybrid predictive modeling.

Pipeline Documentation

1. Data Acquisition & Scraping

• Status: Optional (if raw data is already present).
• Process: Refer to the dedicated Scraping folder to extract live metadata and user reviews directly from the BGG API and web interface.
• Outcome: Generates raw CSV fragments containing disjointed game and user information.

2. Integration & Cleaning

• Process: Scripts in the Merging folder consolidate fragmented batches into a unified master dataset. This dataset is then refined using the high-performance cleaning suite in the Data Cleaning folder.
• Methodology: Includes relational joins, surgical deduplication, and handling of high-cardinality features using memory-efficient chunked processing.
• Outcome: A production-ready file (bgg_data_master_clean.csv) stored in the Data folder.

3. Exploratory Data Analysis

• Process: Located in the Data Analysis folder, these tools provide a statistical and visual overview of the dataset.
• Key Insights: Features performance profiling of prolific users, correlation matrices for game mechanics, and thematic word clouds derived from TF-IDF analysis.

4. Recommender System Implementation

The core architecture is divided into three algorithmic approaches located in the Recommender scripts folder:

1. Collaborative Filtering

• Logic: Utilizes Singular Value Decomposition (SVD) to map latent user-item interactions.
• Validation: Benchmarks performance through individual and global RMSE scores, segmented by user activity levels (Top, Median, and Mean users).

2. Content-Based Filtering

• Logic: Employs a Hybrid Matrix (Numerical Features + TF-IDF) to identify "Mechanical Neighbors."
• Innovation: Implements expansion filtering to ensure recommendation diversity by systematically excluding direct sequels and expansions.

3. Hybrid Recommender System

• Logic: A unified framework integrating Collaborative (SVD) and Content-Based signals through Weighted Hybridization.
• Features:
  • User Stratification: Tools to isolate specific test archetypes such as "Moderate Users" and "Niche Experts."
  • Sensitivity Tuning: An adjustable Alpha ($\alpha$) parameter to balance structural metadata against social community signals.
  • Cold Start Mitigation: Evaluates system resilience against unrated items using content fallbacks.

Getting Started

Installation

Requires Python 3.8+ and the following scientific computing libraries:

Install required libraries: pip install pandas numpy scikit-learn scikit-surprise matplotlib seaborn wordcloud

Execution Order:

• Full Pipeline: Scraping $\rightarrow$ Merging $\rightarrow$ Data Cleaning $\rightarrow$ Data Analysis $\rightarrow$ Recommender scripts.
• Analytical/Modeling Run: Start directly with Data Analysis or Recommender Scripts if a cleaned master file is already present.

Project Structure Overview

• Scraping: Raw data extraction and API harvesting scripts.
• Merging: Batch consolidation and metadata integration logic.
• Data Cleaning: Surgical refinement and memory-optimized deduplication.
• Data: Storage for raw fragments and the finalized production sample dataset.
• Data Analysis: Visual distributions, correlation heatmaps, and thematic profiling.
• Recommender scripts: SVD, Content-Based, and Weighted Hybrid model implementations.