NLP Kaggle — Dialogue Summarization (nanogpt-fudan-CS30040)

A comprehensive record of experiments for the Fudan NLP course dialogue summarization competition , achieving public 0.4485 ( > top-1 0.4477) and private 0.4466 (best).

Competition Overview

• Task: Dialogue summarization (English, SAMSum-style)
• Evaluation: ROUGE (rouge1/rouge2/rougeL)
• Constraint: Model ≤ 400M parameters
• Data: 26,141 train / 2,273 test (dialogue → summary pairs)
• Top-1 baseline: public 0.4477

Final Results

Approach	Public	Private	Notes
MBR-v4 r1	0.4485	0.4437	Best public, exceeds top-1
MBR full-data	0.4455	0.4466	Best private
ProphetNet R-Drop (95%)	0.4393	0.4380	Best single model public
ProphetNet R-Drop (full)	0.4382	0.4407	Best single model private
BART-xsum cossim (95%)	0.4388	0.4314	Best BART single

Approach Evolution

Phase 1: Baseline Models (→0.40-0.43)

Started with straightforward fine-tuning of pre-trained seq2seq models, adding cosine-similarity-based encoder layer pruning to fit the <400M constraint.

Script	Model	Params	Public	Private
plan_a	BART-large prune 2 layers	381M	0.4154	0.4158
plan_c	T5-base	223M	0.3957	0.3978
plan_b	ProphetNet-large	391M	0.4246	0.4286

Key finding: ProphetNet > BART > T5 for dialogue summarization. BART with naive layer pruning underperforms — need smarter pruning.

Phase 2: Refinement (→0.43-0.44)

Improved decoding, truncation repair, and model-specific optimizations.

Script	Change	Public	Private	Δ
plan_a3	BART cossim prune 1 layer, beam6	0.4216	0.4239	+0.008
plan_b2	ProphetNet 5ep, beam6, repair	0.4256	0.4308	+0.002

Key findings:

• Cosine-similarity pruning (remove most redundant layer) outperforms naive layer removal — preserves more quality at same param count.
• Beam search + truncation repair gives small but reliable gains.
• Decoding hyperparameter sweep yields <0.002 variation — not worth tuning.

Phase 3: R-Drop Breakthrough (→0.4380-0.4407)

Applied R-Drop regularization (Liang et al., 2021) — forward pass twice with different dropout masks, add bidirectional KL divergence to loss.

Script	Model	Public	Private	Δ vs no R-Drop
plan_b4	ProphetNet R-Drop 95% data	0.4393	0.4380	+0.007
plan_b4full	ProphetNet R-Drop full data	0.4382	0.4407	+0.003 private

R-Drop on ProphetNet: The single most impactful technique per model. +0.007 on private (0.4308 → 0.4380). alpha=1.0, 5 epochs, fp16 training.

R-Drop on BART-xsum: 4 attempts all diverged (loss → ∞) in epoch 1, across fp16/bf16/fp32 and lr 3e-5/1e-5. Later discovered it works with full data + cossim pruning (see plan_b5full), but the resulting score was lower than no-RDrop. Conclusion: R-Drop benefit is model-specific — works on ProphetNet, not BART-xsum.

Full data vs 95% holdout trade-off: Training on 100% data (no validation holdout) consistently improves private (+0.002-0.003) but hurts public (-0.001 to -0.005). This held for both ProphetNet and BART. Hypothesis: holdout validation provides implicit regularization that helps public distribution generalization; full training overfits slightly to training distribution.

Phase 4: Cross-Model MBR Ensemble (→0.4485)

Minimum Bayes Risk (MBR) decoding: generate N candidates from multiple models, select the one with highest average ROUGE against all other candidates.

Script	Models	Candidates	Metric	Public	Private
mbr_v1	BART+ProphetNet (unbalanced)	10+1	avg ROUGE	0.4378	0.4316
mbr_v2	BART+ProphetNet (balanced)	10+10	avg ROUGE	0.4475	0.4437
mbr_v3	BART+ProphetNet	15+15	avg ROUGE	0.4447	0.4422
mbr_v3_r1	BART+ProphetNet	15+15	rouge1 only	0.4456	0.4447
mbr_v4_r1	BART+ProphetNet	10+10	rouge1 only	0.4485	0.4437
mbr_v4_r2	BART+ProphetNet	10+10	rouge2 only	0.4467	0.4431
mbr_full	BART+ProphetNet (full-data)	10+10	rouge1 only	0.4455	0.4466

Key findings:

1. Balanced candidates are essential: 10+10 (MBR-v2) crushes 10+1 (MBR-v1) by ~0.01. One model dominating candidates biases MBR toward its weaknesses.
2. 20 candidates > 30 candidates: More candidates dilute quality (30 gave 0.4456 vs 20 gave 0.4485). The marginal benefit of additional beams decreases after ~10/model.
3. rouge1 > rouge2 > avg for MBR metric: Evaluating MBR with rouge1-only gives best results. Sweep showed: rouge1 (0.4485) > avg (0.4475) > rouge2 (0.4467).
4. Cross-model diversity matters more than individual model quality: The ensemble of two 0.43-0.44 models produces 0.4485, which neither model can achieve alone. BART-xsum and ProphetNet generate qualitatively different candidates — this diversity is the key ingredient.

Technical Details

Cosine-Similarity Layer Pruning

Used to reduce BART-large-xsum from 406M to 394M parameters.

Method: Forward a batch through the encoder, collect hidden states from each layer, compute pairwise cosine similarity between adjacent layers, remove the layer with highest similarity to its neighbor.

Reference: udnet96/BART-various-finetune (fairseq). We reimplemented in HuggingFace.

R-Drop Regularization

From "R-Drop: Regularized Dropout for Neural Networks" (Liang et al., NeurIPS 2021).

Implementation: Forward each batch twice with different dropout masks. Compute symmetric KL divergence between the two output distributions. Add to cross-entropy loss with weight α=1.0.

loss = cross_entropy + alpha * 0.5 * (KL(p1||p2) + KL(p2||p1))

Works on ProphetNet (+0.007); unreliable on BART-xsum. Hypothesis: BART-xsum's higher dropout rate combined with dual forward pass amplifies gradient variance.

MBR Decoding

For each test sample:

1. Generate N candidates from each model via beam search with num_return_sequences=N
2. For each candidate c, compute average ROUGE-1 against all other candidates
3. Select the candidate with highest average score

Computational cost: O(N² × M) ROUGE computations where M = number of metrics. For 2273 test samples × 20 candidates, this takes ~30 min on CPU.

What We Tried That Didn't Work

Technique	Attempt	Result	Why
R-Drop on BART-xsum	4 attempts	Diverged in epoch 1	High dropout + dual forward
BART-xsum + R-Drop (full data)	1 attempt	Trained but lower score	Cost doesn't justify
30 MBR candidates	1 attempt	0.4456 vs 0.4485 (20)	More dilutes quality
Decoding hyperparameter sweep	6 configs	<0.002 variation	Bottleneck is model, not decode
fairseq cossim BART	~2h effort	Couldn't install	hydra/omegaconf incompatibilities
Full-data training (no holdout)	2 attempts	Private ↑, public ↓	Trade-off, not net win
Average ROUGE for MBR	Multiple	Worse than rouge1-only	rouge1 alone better captures diversity
T5-base	1 attempt	0.3978	Too small for this task

Repository Structure

nlpkaggle/
├── README.md                         # This file
├── ENVIRONMENT.md                    # Environment setup guide
├── .gitignore
│
├── scripts/
│   ├── prophetnet/                   # ProphetNet-based approaches
│   │   ├── plan_b_prophetnet.py      #   Baseline ProphetNet 3ep
│   │   ├── plan_b2_improved_prophetnet.py  # Improved 5ep + beam6 + repair
│   │   ├── plan_b4_rdrop_server.py   # ★ R-Drop ProphetNet (best single public)
│   │   └── plan_b4_fulldata.py       #   R-Drop full-data (best single private)
│   │
│   ├── bart/                         # BART-based approaches
│   │   ├── plan_a_bart_large_prune.py      # Baseline BART prune 2 layers
│   │   ├── plan_a2_improved_bart.py        # Cossim prune 1 layer + beam6
│   │   ├── plan_b5_bart_server.py          # BART-xsum cossim, no R-Drop
│   │   ├── plan_b5base_bart_server.py      # BART-xsum cossim 95% data
│   │   ├── plan_b5_fulldata.py             # BART-xsum full-data + R-Drop
│   │   └── plan_b6_bart_server.py          # BART-large-cnn attempt
│   │
│   ├── mbr/                          # MBR ensemble scripts
│   │   ├── mbr_v3.py                #   30-candidate MBR with metric sweep
│   │   ├── mbr_v4.py                # ★ 20-candidate rouge1 MBR (best public)
│   │   ├── mbr_full.py              # ★ Full-data model MBR (best private)
│   │   └── mbr_decode.py            #   MBR decode utility
│   │
│   ├── t5/
│   │   └── plan_c_t5_base.py        #   T5-base baseline
│   │
│   └── utils/
│       ├── sweep_decode.py          #   Decoding hyperparameter sweep
│       ├── smoke_rdrop.py           #   R-Drop sanity check
│       └── smoke_b5.py              #   BART sanity check
│
├── submissions/                      # Best submission files
│   ├── submission_mbr_v4_r1.csv     # Public 0.4485 (> top-1)
│   └── submission_mbr_full.csv      # Private 0.4466 (best)
│
└── analysis/                         # Detailed analysis documents
    ├── R_DROP_ANALYSIS.md           # R-Drop findings (what worked, what didn't)
    ├── MBR_ANALYSIS.md              # MBR ensemble design space
    └── FULL_VS_HOLDOUT.md           # Full-data vs holdout trade-off analysis

Quick Start

See ENVIRONMENT.md for detailed setup instructions.

Minimal reproduction of best public score:

# 1. Setup environment
python -m venv venv && source venv/bin/activate
pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install transformers==4.44.2 datasets==2.21.0 accelerate==1.14.0 \
            rouge_score==0.1.2 sentencepiece==0.2.0 protobuf==4.25.6 sacrebleu==2.6.0

# 2. Train ProphetNet with R-Drop (→ ckpt_b4/)
python scripts/prophetnet/plan_b4_rdrop_server.py

# 3. Train BART-xsum with cossim pruning (→ ckpt_b5base/)
python scripts/bart/plan_b5base_bart_server.py

# 4. Run MBR ensemble
# Edit mbr_v4.py: BART_DIR=ckpt_b5base/best_model, PROPH_DIR=ckpt_b4/best_model
python scripts/mbr/mbr_v4.py

# 5. Submit submission_mbr_v4_r1.csv to Kaggle

Expected: public ~0.4485 (> top-1 0.4477).

References

• R-Drop: Liang et al., "R-Drop: Regularized Dropout for Neural Networks", NeurIPS 2021
• MBR Decoding: Kumar & Byrne, "Minimum Bayes-Risk Decoding for Statistical Machine Translation", HLT-NAACL 2004
• BART: Lewis et al., "BART: Denoising Sequence-to-Sequence Pre-training", ACL 2020
• ProphetNet: Qi et al., "ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training", EMNLP 2020
• Cosine-similarity pruning: udnet96/BART-various-finetune (GitHub)
• Competition: nanogpt-fudan-CS30040 on Kaggle