Developed and trained entirely on a MacBook M4 Max (64GB RAM, MPS acceleration). Training all three models end-to-end on 36.7 million rows takes approximately 3 hours. The compute constraint forced the same trade-offs production teams face: embedding dimensions, batch sizes, and model complexity must balance quality against memory and latency budgets.
This is a comparative study of three progressively more powerful approaches to click-through rate (CTR) prediction -- the core ML problem behind online advertising, news feeds, and recommendation ranking. The central question: how should we model feature interactions, and how much does the approach matter?
We use the Criteo Display Advertising Challenge dataset -- the industry standard benchmark for CTR prediction. It contains 45.8 million ad impressions with 13 dense (numerical) and 26 sparse (categorical) features, representing real production ad traffic with a 25.6% click-through rate.
The project implements three models that represent distinct philosophies about feature interactions:
- XGBoost -- learns interactions implicitly through tree splits
- DLRM (Meta, 2019) -- learns pairwise interactions explicitly through dot products
- DCN-v2 (Google, 2021) -- learns arbitrary-order interactions through a cross network
CTR prediction is fundamentally about feature interactions. A user's age alone weakly predicts clicks. An ad's category alone weakly predicts clicks. But "25-year-old male AND gaming ad" is a strong signal. The question is how to discover and represent these interactions.
Consider a concrete example with three features: user_age_bucket, ad_category, and time_of_day. A pairwise model (DLRM) can learn "young users click gaming ads" and "evening users click entertainment ads" -- but it cannot directly represent "young users click gaming ads specifically in the evening." That requires a 3rd-order interaction, which DCN-v2's cross network captures naturally.
This is not a contrived example. In production ad systems, the most valuable signals often involve 3-5 features interacting simultaneously: user segment + ad creative + placement + device + time. Models that structurally limit interaction order leave money on the table.
| Model | Paper | How It Models Interactions | Interaction Order |
|---|---|---|---|
| XGBoost | Chen & Guestrin (2016) | Tree splits create axis-aligned interaction regions. Each path from root to leaf is an implicit conjunction of features | Bounded by tree depth (6 by default) |
| DLRM | Naumov et al. (2019) | Embeds each feature into 16-dim space, then computes all pairwise dot products (351 pairs from 27 embeddings) | Exactly 2nd order |
| DCN-v2 | Wang et al. (2021) | Cross layers compute x_0 * (W @ x_l + b) + x_l repeatedly. Each layer adds one interaction order. L layers = up to (L+1)-th order | Up to 4th order (3 layers) |
DLRM (Deep Learning Recommendation Model):
Dense Features (13) Sparse Features (26)
| / | \
Bottom MLP Emb_1 Emb_2 ... Emb_26
(13->64->32->16) (16d) (16d) (16d)
| \ | /
v v v v
[dense_emb] [sparse_emb_1, ..., sparse_emb_26]
\ /
\ /
v v
Pairwise Dot Products (27 choose 2 = 351 values)
|
v
Concat [dot_products, dense_emb] (367 values)
|
Top MLP (367 -> 256 -> 128 -> 1)
|
sigmoid -> P(click)
DCN-v2 (Deep & Cross Network v2):
Dense Features (13) Sparse Features (26)
| / | \
Bottom MLP Emb_1 Emb_2 ... Emb_26
(13->64->32->16) (16d) (16d) (16d)
| \ | /
v v v v
Concatenate ALL (16 * 27 = 432-dim)
|
┌────────────┼────────────┐
v v
Cross Network Deep Network
(3 cross layers) (432->256->128->64)
x_{l+1} = x_0*(W@x_l+b)+x_l
| |
└────────────┬────────────┘
v
Concatenate (432 + 64 = 496)
|
Linear (496 -> 1)
|
sigmoid -> P(click)
| Metric | XGBoost | DLRM | DCN-v2 | Baseline |
|---|---|---|---|---|
| AUC-ROC | 0.7598 | 0.8099 | 0.8110 | 0.5000 |
| LogLoss | 0.5970 | 0.5191 | 0.5313 | 0.6365 |
| PR-AUC | 0.5230 | 0.6145 | 0.6161 | 0.2562 |
DCN-v2 wins on discrimination (AUC, PR-AUC) because it captures higher-order feature interactions that DLRM structurally cannot. The cross network with 3 layers models up to 4th-order crosses -- combinations like "feature_5 AND feature_12 AND feature_19 AND feature_3" that no pairwise dot product can represent. On a dataset with 26 sparse features where multi-way interactions between ad properties, user segments, and context drive clicking behavior, this matters.
DLRM wins on calibration (LogLoss) because its architecture is more constrained. The 351 explicit pairwise dot products create a tighter inductive bias -- there are fewer ways the model can overfit. DCN-v2's full-rank 432x432 weight matrices in each cross layer give it more capacity, which helps discrimination but slightly hurts probability calibration. In production, this would be fixed with Platt scaling or isotonic regression.
XGBoost is significantly behind both neural models despite using 65 engineered features (raw + frequency-encoded + target-encoded). The fundamental limitation: tree models cannot learn embeddings. They treat each categorical value independently, while neural models learn that similar categories (via embedding proximity) should produce similar predictions. With 26 sparse features at up to 100K cardinality each, the embedding approach wins decisively.
The 5-point AUC gap between XGBoost and neural models is massive in production terms. At Meta's scale (billions of ad impressions/day), even 0.1% AUC improvement translates to millions in revenue. The 5-point gap represents a fundamentally different class of model capability.
| Property | XGBoost | DLRM | DCN-v2 |
|---|---|---|---|
| Parameters | ~50K (53 trees) | 11.8M | 12.4M |
| Embedding params | N/A | 11.7M (99%) | 11.7M (94%) |
| Cross/Interaction params | N/A | ~94K (top MLP) | 561K (cross layers) |
| Batch-100 latency | 0.22ms | 1.56ms | 1.06ms |
| GPU required | No | Preferred | Preferred |
| Training time | ~2 min | ~45 min | ~50 min |
DCN-v2 is actually faster than DLRM at inference despite having more parameters. The cross layer computation (matrix multiply + element-wise ops) is highly parallelizable on MPS/GPU, while DLRM's explicit pairwise dot product loop over 351 pairs introduces sequential overhead.
The Criteo Display Advertising Challenge dataset represents real anonymized ad impression logs:
- 45.8 million rows (ad impressions)
- 13 dense features (numerical, anonymized -- likely counters, rates, historical aggregates)
- 26 sparse features (categorical, hashed -- likely ad_id, advertiser, publisher, user_segment, device, placement, etc.)
- Label: clicked (1) or not clicked (0)
- Click-through rate: 25.6% (moderate imbalance, ~3:1 ratio)
Features are anonymized for privacy, but the data characteristics (high-cardinality categoricals, power-law distributions, heavy missing values in dense features) are representative of production ad systems. The vocabulary sizes range from 3 to 100K+ unique values per sparse feature, with most following Zipfian distributions.
| Decision | Constraint | Choice |
|---|---|---|
| Embedding dim = 16 | 26 features x 100K vocab x 16 x 4 bytes = ~160MB. At 64-dim, embedding tables alone exceed 650MB | 16-dim (standard in DLRM paper) |
| Batch size = 8192 | MPS memory budget for forward + backward pass with 12.4M params | 8192 (fills MPS compute units without OOM) |
| 5 training epochs | Diminishing returns after epoch 3-4; each epoch = 4,476 batches on 36.7M rows | 5 epochs with ReduceLROnPlateau |
| Vocab cap = 100K | Long-tail categories below 5 occurrences cannot learn meaningful embeddings | Min frequency 5, max 100K per feature |
| 3 cross layers (DCN-v2) | Each layer adds a 432x432 weight matrix (187K params). Beyond 3 layers, training destabilizes without careful initialization | 3 layers = up to 4th-order interactions |
| Stratified 80/10/10 split | 36.7M train / 4.6M val / 4.6M test. Val set large enough for stable AUC estimation | Standard temporal-agnostic split (no timestamps available) |
notebooks/
01_data_loading_and_exploration.ipynb # Dataset overview, distributions, missing values
02_feature_engineering.ipynb # Log-transform, vocab capping, target encoding
03_xgboost_model.ipynb # XGBoost baseline with 65 engineered features
04_dlrm_model.ipynb # DLRM: embeddings + pairwise dot products
05_dcnv2_model.ipynb # DCN-v2: cross network + deep network
06_evaluation_and_comparison.ipynb # Three-way comparison, calibration, ensembles
models/ # Saved checkpoints, predictions, metrics
data/processed/ # Train/val/test splits (dense, sparse, labels)
plots/ # All generated figures
- PyTorch (MPS backend on Apple Silicon) -- DLRM and DCN-v2 model training
- XGBoost 3.2 -- hist method with early stopping, scale_pos_weight for class imbalance
- Feature engineering -- log1p + standardization for dense features, frequency/target encoding for sparse features, vocabulary capping at 100K
- Evaluation -- AUC-ROC, LogLoss, PR-AUC, ECE (Expected Calibration Error), calibration curves, ensemble analysis
- Class imbalance handling -- BCEWithLogitsLoss with pos_weight (neural), scale_pos_weight (XGBoost)
# Clone the repository
git clone https://github.com/nbatra/ctr-prediction-dlrm-dcnv2.git
cd ctr-prediction-dlrm-dcnv2
# Create environment and install dependencies
uv venv --python 3.13 .venv
uv pip install numpy pandas scikit-learn xgboost torch matplotlib jupyter
# Download Criteo dataset from Kaggle
# Place train.txt in data/raw/ (requires Kaggle account)
# https://www.kaggle.com/c/criteo-display-ads-challenge/data
# Run notebooks in order
.venv/bin/jupyter lab notebooks/Full pipeline (all 6 notebooks) takes approximately 3 hours on an M4 Max. The majority of time is spent on neural network training (notebooks 04 and 05, ~50 min each). XGBoost trains in under 2 minutes.
The data/ and models/ directories are excluded from version control due to size (~35GB raw dataset, ~200MB trained artifacts). To regenerate:
- Seed data: Download the Criteo Display Advertising Challenge dataset from https://www.kaggle.com/c/criteo-display-ads-challenge/data (requires Kaggle account). Place
train.txt(11GB, 45.8M rows) indata/raw/. - Processed features: Notebooks 01-02 perform feature engineering (log-transform, vocab capping, target encoding, train/val/test split) and save processed parquets in
data/processed/. - Models: Training notebooks generate model checkpoints in
models/:- NB03 trains the XGBoost baseline (53 trees, ~2 min)
- NB04 trains the DLRM model (embeddings + dot-product interactions, ~45 min)
- NB05 trains the DCN-v2 model (cross network + deep network, ~50 min)
-
Feature interactions are where the signal lives in CTR prediction. All three models achieve their gains by capturing interactions -- they just differ in how explicitly and to what order. The 5-point AUC gap between XGBoost (implicit interactions via tree splits) and DCN-v2 (explicit cross network) demonstrates that structured interaction modeling matters.
-
Embeddings are the decisive advantage of neural CTR models. 94-99% of parameters are in embedding tables. These learned representations allow the model to generalize across similar categorical values (similar ads, similar users) -- something tree-based models fundamentally cannot do with high-cardinality features.
-
Higher-order interactions provide marginal but consistent gains over pairwise. DCN-v2 beats DLRM by ~0.1% AUC. This is modest on a single dataset, but the gap widens with richer feature sets and at production scale. The cross network's advantage is architectural: it can represent interactions DLRM structurally cannot.
-
Calibration and discrimination are separate concerns. DLRM has the best LogLoss despite lower AUC than DCN-v2. In production, ranking (AUC) and bid pricing (calibration) are different use cases. A model that ranks well but predicts poorly-calibrated probabilities needs post-hoc calibration before it can be used for auction pricing.
-
Ensemble diversity exists between model families. Trees, pairwise neural, and cross-network neural models fail on different subsets of data. A simple average of all three outperforms any individual model, confirming they capture complementary signal. Production systems routinely ensemble different architectures for this reason.
Built by Nipun Batra
This project is released for educational and portfolio purposes. The Criteo dataset is provided by Criteo Labs under their own terms of use.
Keywords
Click-Through Rate, CTR Prediction, DLRM, Deep Learning Recommendation Model, DCN-v2, Deep & Cross Network, Feature Interactions, Embedding Tables, XGBoost, Criteo Dataset, Online Advertising, Binary Classification, Dot Product Interactions, Cross Network, Higher-Order Features, Production ML, Recommendation Systems, AUC-ROC, LogLoss, Calibration, Platt Scaling, Ad Tech, Programmatic Advertising, Real-Time Bidding, Sparse Features, Dense Features, Model Ensemble, PyTorch, Gradient Boosting