Mastercard Data Quest — Hidden Micro-Businesses

Detect consumer cardholders whose transaction behavior looks like a business, so the bank can offer them SME / acquiring / lending products.

Documents

• CASE.md — the official case brief and scoring rubric.
• ABOUT_MASTERCARD.md — company context and what judges expect.
• STRATEGY.md — full methodology (consolidated from the planning PDFs) + the data-grounded revised plan.
• DATASET_ANALYSIS.md — real-data findings, the personal-vs-business ratio test, and the leakage warning.

Setup (teammates read this first)

The dataset is not in the repo (gitignored — too large). Before running:

1. Obtain the 3 parquet files and place them in dataset/ (see dataset/README.md for names/schema).
2. pip install -r requirements.txt (on macOS LightGBM also needs brew install libomp).

artifacts/ (features, models, ranked lists) is gitignored too — it is created locally when you run, not cloned.

Pipeline

pip install -r requirements.txt              # needs libomp for lightgbm: brew install libomp

# One-command best-approach pipeline (data -> features -> eval -> ranked list):
python main.py                               # uses ./dataset, best method (PU-learning)
python main.py --data-dir path/to/data       # run on any dataset with the same schema
python main.py --method m1_supervised        # switch detector

Per-method / research runs (from src/):

cd src
python -m common.build_features              # dataset/*.parquet -> artifacts/features.parquet
python -m methods.m4_pu_learning.run         # best method -> artifacts/m4_pu_learning/
python -m evaluation.compare                 # benchmark all methods (inject-test)
python -m viz.make_figures                   # regenerate presentation figures

Structure (one folder per detection method; see docs/methodology/00_overview.md):

• src/common/ — shared: mcc_categories.py, features.py, build_features.py, base.py (method interface), metrics.py, ranking.py.
• src/methods/m1_supervised/ — Method 1 (supervised LightGBM).
• src/methods/m2_one_class/, m3_similarity/, m4_pu_learning/, m5_rule_based/ — the other detection methods.
• src/combinations/ — ensembles that combine methods (rank-average, gating).
• src/evaluation/ — inject_test.py + compare.py (benchmark all methods).
• src/viz/make_figures.py — generates presentation figures.
• presentation/ — PRESENTATION.md (deck pack) + figures/ (committed PNGs) + web/ (Vite + React deck app, bilingual RU/EN).

Interactive dashboard (backend/ + frontend/)

A live dashboard over the model: searchable ranked candidate list → click a card → its transactions + score + reasons + per-feature SHAP explanation, plus data-level analytics (score distribution, tiers, segments, top merchants, spend over time, business-vs-consumer comparison). Stack: FastAPI + Postgres (Docker) with a separate Vite/React frontend (deploys to Vercel).

docker compose up -d db                      # Postgres
docker compose run --build --rm etl          # one-shot: parquet -> features -> score -> SHAP -> Postgres
docker compose up -d api                      # API on :8000
cd frontend && npm install && npm run dev     # UI on :5173

If the Docker VM is memory-constrained, run the ETL on the host (same script): cd backend && DATABASE_URL=postgresql://mdq:mdq@localhost:5432/mdq REPO_ROOT=$(pwd)/.. MODELS_DIR=$(pwd)/models PYTHONPATH=$(pwd)/../src python -m etl.build. See backend/README.md and frontend/README.md. The dashboard refits the documented PU-learning model (src/methods/m4_pu_learning) on the 31 features; the boosters under native_production_models/ use an orphaned 24-feature schema and are not used.

Docs (docs/):

• docs/case/ — the given case material: CASE.md, ABOUT_MASTERCARD.md, original PDFs.
• docs/analysis/ — our analysis: STRATEGY.md, DATASET_ANALYSIS.md, RESULTS.md.
• docs/methodology/ — method deep-dives + results, audit, optimization, roadmap.
• docs/ru/ — Russian explainers (ОБЪЯСНЕНИЕ, УЛУЧШЕНИЯ, ЧТО_СМОТРИТ_МОДЕЛЬ).

Key findings

• Data is spend (outflow) only — no senders/inflows. All "unique sender" features from the original plan are impossible; the signal is where/how a card spends.
• card_tier is leakage (100% "Business" in the business file) — excluded.
• Personal-vs-business spend ratio is the core signal: business cards spend 4% personal / 85% business; consumers 67% / 0.3%. A single feature separates the classes at AUC ≈ 0.96.
• Supervised separation is near-perfect (AUC ≈ 1.0) because the data is synthetic — so the train metric is not the story. The deliverable is the ranked list of hidden businesses among the 80k consumers, validated by sustained B2B behavior (recurring spend, repeat wholesalers, large tickets, low personal spend).

artifacts/ is gitignored — derived and reproducible from the steps above.