This repository contains the code, documents, and experiment infrastructure for my Colorado School of Mines master's project, completed during Fall 2025 and Spring 2026 in collaboration with Dr. Bo Wu.
This project studies user-level downstream personalization as a decision-prediction problem. Instead of asking whether a model can imitate a user's writing style, the core question is whether it can predict how a specific Manifold trader will act in a given market using only information available before that decision. Manifold prediction markets provide a useful setting for this because user behavior is sequential, timestamped, and often accompanied by additional reasoning signals such as comments and stake sizes.
The repository contains the full research stack behind that question: data ingestion from the Manifold API, PostgreSQL schemas and normalization, pgvector embeddings, internal APIs, LoRA training and inference services, and experiment code for prompting, retrieval, per-user adapters, and tool-using agents. The benchmark compares four main strategies: a random-sampling baseline, an LLM prompting baseline, per-user LoRA adapters, and an agent-based approach.
The final report's main result is that the agent-based method performed best in this setting, reaching 67% accuracy versus 48% for per-user adapters, while also using roughly 10x fewer tokens and running about 4x faster when training and inference costs are considered together. This README is meant to explain what the project is and how the repository is organized, not to serve as a turnkey reproduction guide.
- Final report (PDF)
- Final report source
- Project presentation (PDF)
- Presentation source
- Project proposal (PDF)
- Proposal update (PDF)
- Persistent adapter training artifacts
- Persistent adapter experiment logs
- Persistent adapter experiment summary data
manifold_common/- shared SQLAlchemy schemas and typed models for users, bets, contracts, comments, embeddings, contract answers, and strong-user tables.manifold_etl/- Manifold API extraction, normalization, staging and public loaders, DB bootstrap SQL, snapshots, merge utilities, and embedding backfill scripts.manifold_api/- internal FastAPI service for authenticated reads over normalized Postgres data plus embedding-related routes.gpu_api/- FastAPI GPU worker for LoRA training and inference, adapter lifecycle management, checkpointing, and worker service scripts.manifold_experiments/- main experiment suite and CLI for the personalization benchmark.manifold_experiments/common/- shared experiment runner, logging, metrics, ground-truth construction, LLM wrappers, and user-context helpers.manifold_experiments/rag/- retrieval-based prompting experiment.manifold_experiments/main_experiment/- comparison run that coordinates baseline, RAG, LoRA, and agent methods.manifold_experiments/agent/- agent runtime, prompt construction, compaction, and tool API client code.manifold_experiments/agent_api/- constrained FastAPI tool layer over Postgres and embeddings for agent experiments.manifold_experiments/agent_experiment/- agent-only evaluation pipeline.manifold_experiments/adapter_params/- per-user adapter hyperparameter sweep code.manifold_experiments/persistent_training/- durable per-user training runs, artifact storage, split handling, and validation utilities.manifold_experiments/persistent_experiment/- evaluation and comparison code for persisted adapters and related artifacts.manifold_experiments/data/- run summarization, graph generation, and strongest-user analysis scripts.documents/project_report/- final written report, figures, bibliography, and LaTeX sources.documents/project_presentation/- final slide deck and LaTeX sources.documents/project_proposal/- proposal documents and bibliography sources.documents/ml_proposal/- earlier proposal-stage baseline scaffold and notes for a simple-feature ML prototype.
Note: the experiments package now lives under manifold_experiments/. Commands, imports, and docs should use that exact path and module name.