Skip to content

Add Ministral3 DeepSpec reproduction experiments#50

Draft
andylolu2 wants to merge 49 commits into
deepseek-ai:mainfrom
andylolu2:andy/ministral-3b-repro
Draft

Add Ministral3 DeepSpec reproduction experiments#50
andylolu2 wants to merge 49 commits into
deepseek-ai:mainfrom
andylolu2:andy/ministral-3b-repro

Conversation

@andylolu2

@andylolu2 andylolu2 commented Jul 4, 2026

Copy link
Copy Markdown

Why

This experiment checks whether the DeepSpec Eagle3 and DSpark recipes reproduce paper-level accepted-length results when adapted to mistralai/Ministral-3-3B-Instruct-2512. The reports are intentionally included even though the result is negative: the implementation and eval pipeline work, but the Ministral3 runs do not match the Qwen3-4B reference behavior.

Summary

  • Add Ministral3 Eagle3 model/trainer/evaluator support and 10k Open-PerfectBlend reproduction scripts.
  • Add save-only checkpoints, incremental eval JSON writing, split eval task launching, and plotting utilities.
  • Add DSpark block-7 Ministral3 follow-up launchers, arena-only rescue eval launchers, and a validated eval JSON merge helper.
  • Add final markdown reports with training curves and accepted-length comparison plots.

Reviewer guide

  • Real logic changes: deepspec/modeling/eagle3/ministral3, deepspec/trainer/eagle3_trainer.py, deepspec/eval/base_evaluator.py, eval.py, and checkpoint handling in deepspec/trainer/base_trainer.py.
  • Experiment/config changes: config/eagle3/eagle3_ministral3_3b.py, experiments/ministral3_3b_eagle3/*.sbatch, and experiments/ministral3_3b_followup/*.sbatch.
  • New follow-up utility: experiments/ministral3_3b_followup/merge_eval_json.py validates and merges a single-task rescue eval into a complete aggregate JSON.
  • Report artifacts: experiments/ministral3_3b_eagle3/report.md covers the Eagle3 baseline reproduction; experiments/ministral3_3b_followup/README.md covers the completed DSpark follow-up. These reports are the best entry points for reviewing the outcome.

Test plan

  • Ran Python syntax checks for eval.py, the follow-up plotting/merge helpers, and shell syntax checks for split eval sbatch scripts.
  • Ran smoke data/cache/train jobs for Ministral3 support before full runs.
  • Completed the 3000-step DeepSpec Eagle3 training run and final 9-task eval. Final eval JSON: /mnt/vast/runs/andy/deepspec_ministral3_3b_eagle3/eval_10k_step3000_metrics.json.
  • Completed the 3000-step DSpark block-7 training run and final 9-task eval. Final eval JSON: /mnt/vast/runs/andy/deepspec_ministral3_3b_followup/eval_dspark_8gpu_step3000_metrics.json.
  • Re-ran arena-hard-v2 as a one-task rescue eval after the full DSpark eval timed out at 8/9 rows, then validated and merged it with merge_eval_json.py.
  • Regenerated report plots from TensorBoard and the completed eval JSONs.

@andylolu2 andylolu2 changed the title Add Ministral3 Eagle3 reproduction experiment Add Ministral3 DeepSpec reproduction experiments Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant