Market ML Portfolio

Reproducible machine-learning research on 1-minute OHLCV stock data for 44 Nasdaq-100 tickers spanning Nov 2020 – Dec 2025 (≈ 21.2 million bars). The portfolio demonstrates rigorous data science, feature engineering, dimensionality reduction, deep learning, leakage-safe validation, experiment tracking and clean, testable code — not a profitable trading strategy.

---

1. Objective

Build a complete, end-to-end ML research pipeline on high-frequency financial time-series that an employer can clone and reproduce. Every step is engineered to prevent look-ahead bias, every result is reported honestly, and every component is modular.

2. Data

Source	SQLite historical_data_5y.db (3.4 GB), table ohlcv_1min
Tickers	44 (AAPL, ABNB, ADBE, …, TSLA, TXN, VRTX + QQQ ETF)
Coverage	2020-11-18 → 2025-12-05, 1-minute resolution, US regular hours (09:30–15:59 ET)
Rows	21,209,393 raw / 19,554,203 after feature NaNs
Schema	symbol, datetime (tz-aware), open, high, low, close, volume

A per-ticker data-quality report is generated at reports/results/data_quality.csv (missing in-session bars, zero-volume bars, OHLC sanity violations, return outliers, gaps).

3. Repository layout

market-ml-portfolio/
├── configs/config.yaml              single YAML controlling every script
├── scripts/                         entry-points (P1 → P5)
├── src/
│   ├── data/        loader, SQLite I/O, validation, resampling
│   ├── features/    leakage-safe features + forward targets
│   ├── models/      sklearn baselines, CNN, RNN/LSTM/GRU, window builder
│   ├── evaluation/  classification & regression metrics
│   ├── visualization/ shared matplotlib helpers
│   ├── utils/       config, logging, splits
│   └── app/dashboard.py  Streamlit dashboard
├── tests/           pytest – feature leakage, split disjointness, validators
├── reports/
│   ├── figures/     EDA + confusion + calibration + training curves
│   ├── results/     CSV / JSON metrics
│   └── model_cards/ per-project model cards
├── .github/workflows/ci.yml   lint + tests on every push
└── requirements.txt

4. The six projects

Each project has its own step-by-step walkthrough with reasoning: Project 1 · Project 2 · Project 3 · Project 4 · Project 5 · Project 6

Project 1 — Data engineering & feature store

End-to-end pipeline ingesting all 44 tickers from SQLite and producing a per-symbol Parquet feature store and a long-format panel.

• 62 features per bar across 8 families: returns (log/simple, multi-horizon), volatility (rolling std, realized vol, ATR, HL-range), volume (rolling mean, relative volume, z-score, dollar volume), trend (SMA/EMA distance, MACD), momentum (RSI, stochastic, ROC), intraday clock (min-of-day, hour, day-of-week), VWAP distance, opening-range break, plus cross-sectional percentile ranks across the universe at each timestamp.
• A separate targets.py module produces forward-looking labels (30-min and 60-min log return, direction, big-move flag, vol-breakout flag, daily direction).
• Strict leakage prevention: every rolling/EWM uses trailing windows; opening-range values are only valid after the OR window has fully elapsed; cross-sectional ranks use only data available at the bar.

Outputs: data/processed/panel.parquet (21.2M × 71), reports/results/data_quality.csv, reports/results/feature_dictionary.json, 4 EDA figures.

Project 2 — PCA + classical baselines

• Standardized 51 numerical features fit on training rows only.
• PCA fit on training: PC1–10 explain 83.6 %, 14 PCs reach 90 % of variance.
• Six classifiers compared on the same time-ordered split, predicting next-30-minute direction:

Model	ROC-AUC	F1	Accuracy	Brier
Majority class	0.5000	0.666	0.499	0.250
Momentum rule (sign of ret_log_30)	0.495	0.495	0.496	0.283
Logistic regression	0.506	0.547	0.504	0.250
Random forest (300 × depth 10)	0.505	0.546	0.503	0.250
XGBoost (400 × depth 6)	0.502	0.520	0.500	0.251
LightGBM (600 × 63 leaves)	0.503	0.520	0.501	0.251
PCA(5) + logreg	0.507	0.575	0.504	0.250
PCA(20) + logreg	0.508	0.550	0.505	0.250
PCA(40) + logreg	0.506	0.547	0.504	0.250

(2 M train, 0.5 M val, 0.5 M test rows, sampled within the chronological splits.)

Project 3 — Conv1D CNN for short-window classification

• Causal Conv1D blocks (32 → 64 → 64) + BatchNorm + Dropout + global average pooling + dense head with sigmoid output.
• Inputs: 60-minute rolling windows of 16 normalized features, stride 15, 730 K windows built across the full universe, of which 600 K sampled for training.
• Class-weighted binary cross-entropy, Adam, ReduceLROnPlateau + EarlyStopping on val ROC-AUC.

Project 4 — Recurrent networks (SimpleRNN / LSTM / GRU)

• Two-layer stacks (units → units/2) with LayerNorm on inputs and dropout — identical window scheme and training infra as Project 3 for an apples-to-apples comparison.

Deep-model test-set results (next-30-min direction, full universe):

Architecture	ROC-AUC	PR-AUC	Accuracy	F1	Brier
CNN	0.496	0.497	0.497	0.558	0.251
SimpleRNN	0.499	0.499	0.500	0.496	0.251
LSTM	0.497	0.499	0.498	0.537	0.250
GRU	0.500	0.501	0.500	0.490	0.250

Holdout 2025+ numbers are tracked alongside in reports/results/p34_deep_models.csv and via MLflow.

Honest verdict: none of the deep models meaningfully beats the majority baseline on next-30-min direction. This is the expected outcome given the absence of order-book / trade-tape micro-structure features, and it is reported transparently. The portfolio's value is the engineering quality — not a hindsight-fit AUC number.

Project 5 — Experiment tracking and governance

• Every run from scripts/03_train_deep.py is logged to MLflow (mlruns/) with parameters, metrics, training history, saved Keras models, and feature-column manifests.
• TensorBoard logs are emitted to logs/tensorboard/<arch>/.
• The whole pipeline is YAML-config-driven (configs/config.yaml).
• 6 pytest tests cover: validation, OHLC sanity cleaning, feature non-look-ahead invariance, forward-target offset correctness, split disjointness, walk-forward fold ordering.
• GitHub Actions workflow runs ruff + pytest on every push (.github/workflows/ci.yml).
• Per-project model cards in reports/model_cards/.

Project 6 — Streamlit dashboard

Interactive app (src/app/dashboard.py) with five tabs: candlestick chart, engineered-feature overlays, PCA variance, model comparison tables, per-ticker data-quality report.

PYTHONPATH=. streamlit run src/app/dashboard.py

---

5. Methodology highlights

• Splits. Train ≤ 2023-06-30, Val 2023-06-30 → 2024-06-30, Test 2024-06-30 → 2025-06-30, Forward holdout 2025-06-30+. Embargoed by the maximum target horizon at every boundary.
• Leakage prevention is the central design principle: see src/features/build.py (every feature uses trailing data only), src/features/targets.py (forward labels in a separate module), src/utils/splits.py (chronological splits with embargo), and tests/test_pipeline.py::test_features_no_lookahead.
• Standardization fit only on training rows in scripts/02_pca_and_baselines.py; per-symbol z-scoring uses each symbol's training slice only in src/models/windows.py.
• No row shuffling across time. Windows can be shuffled inside the training split because each window is itself causal.

6. How to run

# 1. install
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. point configs/config.yaml at your SQLite DB if different

# 3. build the feature panel for the full 44-ticker universe (~13 min)
PYTHONPATH=. python scripts/01_build_features.py

# 4. PCA + classical baselines (~7 min, includes RF + XGB + LGBM + 4 PCA-LR variants)
PYTHONPATH=. python scripts/02_pca_and_baselines.py

# 5. CNN + SimpleRNN + LSTM + GRU on full universe, tracked in MLflow (~12 min on M-series CPU)
PYTHONPATH=. python scripts/03_train_deep.py

# 6. tests
PYTHONPATH=. pytest -q

# 7. dashboard
PYTHONPATH=. streamlit run src/app/dashboard.py

# 8. MLflow UI
mlflow ui --backend-store-uri ./mlruns

7. Key skills demonstrated

Python · pandas · numpy · scikit-learn · TensorFlow / Keras · Conv1D CNN · LSTM / GRU / SimpleRNN · PCA · time-series feature engineering · walk-forward & chronological validation · classification + regression metrics with calibration · XGBoost · LightGBM · Random Forest · MLflow experiment tracking · TensorBoard · pytest · GitHub Actions CI · Streamlit · Plotly · YAML-driven configs · model cards / data cards

8. Limitations

• Single data source (consolidated tape OHLCV); no order book, no trade tape, no fundamentals.
• No transaction-cost / market-impact modeling — would gate any “does this make money” question.
• Models are tuned at modest scale on CPU; capacity, regularization and architecture sweeps were intentionally kept simple to keep the project reproducible end-to-end in under 30 minutes.
• The next-30-minute direction target is dominated by noise; future work would re-frame as risk-bucket or vol-regime classification (already supported by src/features/targets.py).

9. Future work

• Add explicit walk-forward retraining loop (folds already implemented in src/utils/splits.py::walk_forward_folds).
• Replace majority-vote target with quintile-bucketed forward returns to recover signal.
• Try Transformer / TCN / Temporal Fusion Transformer on the same window dataset.
• Add SHAP explanations to the dashboard for the tree models.
• Dockerize for one-line reproduction.

10. Positioning

I built a reproducible machine-learning research pipeline for high-frequency financial time-series data, including feature engineering, dimensionality reduction, deep-learning architectures, leakage-safe validation, experiment tracking, and model interpretation — and reported the results honestly.

That is what this repository is.