Skip to content

PROGRESS.md

Latest commit

 

History

History
41 lines (30 loc) · 10.4 KB

PROGRESS.md

File metadata and controls

41 lines (30 loc) · 10.4 KB
Raw

Medea — Progress

One line per milestone. Update when finishing a milestone.

  • M1 — Bootstrap: ✅ done (2026-04-28) — project layout, deps installed via uv sync (507 pkgs), GPU verified
  • M2 — Ingest: ✅ done (2026-04-28) — 94 clips (44 offender / 50 control) at 30s middle slices, ≤720p, in data/raw/. SQLite at data/medea.db (channels + videos)
  • M3 — Visual + audio features: ✅ done (2026-04-29) — data/features/visual.parquet (94×512 CLIP ViT-B/32 mean-pooled, L2-normed) and data/features/audio.parquet (faster-whisper transcript + lang + wav2vec2 ai_voice_prob). Strong class signal already: offender ai_voice median 0.86 vs control 0.37; offender mean transcript 123 chars vs control 303.
  • M4 — Text + metadata features: ✅ done (2026-04-30) — data/features/text.parquet (94×384 transcript + title+desc MiniLM embeddings, L2-normed, zero vec for empty transcripts) and data/features/metadata.parquet (94×15 handcrafted scalars: title shape/caps/clickbait/emoji, desc url+hashtag counts, view_count_log, channel age + mean inter-upload). Fused into data/features/combined.parquet shape (94, 1296) = 512 visual + 384 transcript + 384 title_desc + 16 scalars. New class signals: desc_url_count median control=3 vs offender=0; view_count_log median 5.1 vs 3.9.
  • M5 — Vector DB + UMAP: ✅ done (2026-04-30) — Chroma persistent collection videos at data/chroma/, populated with the 1280-dim embedding-only slice (visual+transcript+title_desc; scalars omitted to keep cosine geometry meaningful). UMAP rendered to data/features/umap.png and notebooks/01_explore_embeddings.ipynb: 10 distinct channel clusters (channel-level leakage is real — M6 holdout must split by channel), but offender/control halves are visibly separated on UMAP-1. Channel-level LOO kNN (k=5, cosine-weighted): acc=0.81, prec=0.86, rec=0.71, F1=0.78 — already over the M6 bar of 0.7 precision before any classifier is trained. Hard cases: ch1 (offender, ASMR-shorts) → 1/9; ch20 (offender, only 5 videos) → 0/5; ch72 (control) → 6/10.
  • M6 — LogReg baseline: ✅ done (2026-04-30) — StandardScaler → LogisticRegression(C=1.0, class_weight='balanced') on the full 1296-dim fused vector, evaluated with LeaveOneGroupOut over channel_id (10 folds). Out-of-fold metrics: acc=0.809, prec=0.783, rec=0.818, F1=0.800, ROC-AUC=0.934. Plan's precision-≥-0.7 bar cleared. Vs M5 kNN: F1 0.78→0.80, recall 0.71→0.82 (LogReg moved the boundary toward catching more offenders), precision 0.86→0.78. Per-channel: improved on offender failures (ch1 0.11→0.44, ch20 0.00→0.40, ch72 0.60→0.70) but regressed on ch61 ctrl (0.90→0.30, mean_score 0.59 — genuine confusion, not threshold). Final pipeline saved to data/models/logreg.joblib (refit on all 94 samples); mlflow run logged under data/mlruns/medea-baseline/.
  • M7 — MLP classifier: ✅ done (2026-04-30) — small PyTorch MLP (1296→128→64→1 with LayerNorm + GELU + Dropout=0.4), AdamW(lr=1e-3, wd=1e-3), BCEWithLogitsLoss with pos_weight=neg/pos, batch=16, max 200 epochs. Same outer LeaveOneGroupOut(channel_id) as M6, plus an inner channel-level val split (1 train channel held out per fold) for early stopping (patience=25). Out-of-fold metrics: acc=0.872, prec=0.833, rec=0.909, F1=0.870, ROC-AUC=0.933. Plan's "non-trivial F1 margin over LogReg" bar met (0.80→0.87 = +0.07). Per-channel improvements over LogReg: ch1 0.44→0.67, ch20 0.40→0.80, ch72 0.70→0.90. ch61 ctrl still stuck at 0.30 (mean_score 0.65) — both LogReg and MLP fail on it the same way; flagged for M9 error analysis. Final model refit on all 94 for median best epoch (150) and saved to data/models/mlp.pt (state_dict + scaler stats + arch config); mlflow run logged under medea-baseline/mlp-baseline.
  • M8 — Predict CLI + FastAPI: ✅ done (2026-04-30) — medea.model.infer.Predictor is the single source of truth for inference: URL → 30s clip → 4-modality features → fused vector (built byte-for-byte the same as features.pipeline) → score + top-k neighbors + LogReg-surrogate attribution. CLI at scripts/predict.py <url> [--model mlp|logreg] [--k N] [--json]; FastAPI at medea.api.server:app exposing POST /predict and GET /health, with the MLP predictor warmed up in the lifespan handler. Bug found and fixed: at inference we have only 1 video per channel, but training channels had 5–10. The three channel scalars (count, age, mean inter-upload) became multi-sigma OOD inputs that drove a spurious "AI" signal — fixed by substituting the StandardScaler's training-set mean for those columns at inference (z-score → 0, contribution → 0). Verified end-to-end on 3 in-sample clips (P(AI)=0.000/1.000/1.000 as expected) and one fresh URL (Tom Scott "Diggy Diggy Hole"). MLP-vs-LogReg disagreement on OOD music video content: MLP=0.997, LogReg=0.345 — the MLP, despite winning in CV, extrapolates much more aggressively. Flagged for M9.
  • M9 — Error analysis (first iteration): ✅ done (2026-04-30) — built scripts/error_analysis.py (channel-grouped LOGO CV with both models, OOF preds cached at data/features/oof_predictions.parquet, FP/FN tables with titles, ch61 deep-dive, MLP-vs-LogReg disagreement OOD proxy). Root cause for ch61: it's Primitive Technology — silent wilderness builder, no speech. Median ai_voice_prob=0.815 (offender-range) because the wav2vec2 anti-spoof model returns ~noise on silent clips. Same trap caught the ch1 ASMR offenders (0% has_speech) → top FNs. Fix: added two scalar features to pipeline.py: has_speech (transcript non-empty) and transcript_chars. Scalar block grew 16 → 18; fused vector 1296 → 1298. Result (MLP): F1 0.870 → 0.901, ROC-AUC 0.933 → 0.961, ch61 0.30 → 0.60, ch1 0.67 → 0.89. ch20 regressed 0.80 → 0.60 (n=5, noisy). LogReg unchanged (F1=0.800) — the "use ai_voice_prob only when has_speech=1" interaction is multiplicative, can't be expressed linearly. Tom Scott OOD case still bad (MLP=0.994, LogReg=0.366) — that's a different problem (extrapolation outside training cone), not feature-space.
  • M9 — Error analysis (second iteration, temporal prior): ✅ done (2026-04-30) — added an inference-side P(AI) cap in medea.model.infer._temporal_prior_cap keyed on VideoMeta.upload_date. Pure prior, not a learned feature: training data is 2024–2026 only so the model couldn't learn "old → human" on its own. Cutoffs: pre-2017 → 0.05; 2017–2019 → 0.30; 2020–2021 → 0.65; 2022+ → uncapped (DALL-E 2 / Stable Diffusion / Modelscope era). min(model_score, cap) — never raises confidence, only bounds it. Resolves Tom Scott OOD failure: 2014-07-11 upload, raw MLP=0.994 → final P(AI)=0.05. Prediction dataclass now carries raw_score, upload_date, prior_cap; CLI surfaces "temporal prior applied" message; FastAPI response and JSON output include the new fields.
  • Stretch — local-LLM RAG with rule-based fallback: ✅ done (2026-04-30) — medea.model.explain has two render paths sharing the same retrieved neighbor context (top-k Chroma neighbors hydrated with titles + transcript heads from SQLite + audio.parquet): (1) _render_llm calls Ollama on localhost:11434 (default qwen2.5:7b-instruct-q4_K_M, override via MEDEA_OLLAMA_MODEL) — a true RAG path with no API keys; (2) _render_rules is the deterministic neighbor-grounded template. explain() tries the LLM first, catches any failure (daemon down, model not pulled, generation error) and falls back to rules with a [rule-based; ollama not running] marker. Verified end-to-end with Ollama daemon running: 7B produces honest rationales citing real handles from SQLite. Tried 3B first (llama3.2:3b) — too weak for this task: hallucinated DISAGREEMENT lines, contradicted the verdict, mis-attributed the temporal prior to "training time." Tightened system prompt + pre-rendered verdict labels improved 3B but didn't solve it; 7B fixed the issue. Default bumped to 7B; on RTX 3080 10GB it fits with our pipeline (~3GB used) plus margin.

Next action when resuming

M9 is open-ended ("ongoing" per plan). The first iteration just landed (has_speech + transcript_chars). Three concrete leads remain, in rough priority order:

  1. OOD calibration of the MLP. Tom Scott music video: MLP=0.994 vs LogReg=0.366 on the same vector — large gap unchanged by has_speech. The MLP extrapolates aggressively on content unlike training. Two paths: (a) widen the seed list (more channels, more genres — directly attacks the small training cone) — likely highest-leverage; (b) temperature scaling of the MLP logits on a held-out validation slice for cheap calibration. Try (a) first.

  2. ch20 regressed at M9 (0.80 → 0.60). Only n=5 videos so noise dominates, but worth checking whether one specific video flipped. Cheap to diagnose with error_analysis.py --refresh and grepping for ch20.

  3. Channel-scalar inference shim → real fetch. medea.model.infer._NEUTRALIZED_CHANNEL_COLS substitutes training means; the right fix is list_channel_videos(channel_url, n=10) at predict time → real upload dates → real cadence stats. Costs ~10 round-trips per prediction; correctness vs latency trade-off.

Optional stretch — the Stretch goal in PLAN.md: RAG explanation. Take the top-3 Chroma neighbors (already computed by Predictor), feed their transcripts + the model's score to Claude, generate a short "why this looks AI" rationale. Cleanly slots into the existing Predictor output.

Useful entry points (unchanged):

  • medea.model.infer.Predictor — same path CLI/API/error analysis go through.
  • python scripts/extract.py --modality combined — rebuilds combined.parquet.
  • python scripts/train.py && python scripts/train_mlp.py — re-runs both classifiers.
  • python scripts/build_chroma.py — repopulates Chroma.
  • python scripts/error_analysis.py [--refresh] — re-runs CV and prints OOF error tables; cached at data/features/oof_predictions.parquet.

Notes

  • Plan file: PLAN.md.
  • Project rules: no git/commits per user preference for local practice projects. No third-party API keys / external API integrations — confirmed at the M9 stretch when the LLM-API rationale was reverted in favor of a local rule-based version.
  • Hardware: RTX 3080 10GB, Python 3.12.10, Windows 11.
  • ffmpeg is bundled via imageio-ffmpeg and copied as data/.bin/ffmpeg(.exe) so yt-dlp finds it; it also gets prepended to PATH at runtime.