Welcome! 👋
This repository contains 160+ pages of handwritten notes that I created while taking Stanford CS336: Language Modeling from Scratch (Spring 2024).
The course, instructed by Percy Liang and Tatsunori Hashimoto, provides a comprehensive introduction to the principles and systems behind modern Large Language Models (LLMs), covering topics from pre-training datasets, tokenization, transformer architectures, LLM pretraining to distributed training, inference and LLM serving, memory optimization, GPU benchmarking and profiling, collective communcation, AI evaluation, and RL alignment and policy optimization, ect.
My goal was not simply to pass the course, but to build a deep understanding of how LLMs are developed from the ground up.
Throughout the course, I handwrote detailed notes to reinforce my understanding of the mathematical foundations, implementation details, systems concepts, and engineering trade-offs. Looking back, these notes played an important role in shaping my knowledge of:
- 🧠 Transformer core principles (attention, positional encoding, architecture design)
- 🔤 Tokenization and vocabulary construction (BPE, SentencePiece)
- ⚙️ PyTorch fundamentals and tensor ops (einops) for LLM implementation
- 🧹 Dataset preparation, cleaning, filtering, and corpus engineering
- 🧪 Data-centric techniques (data mixing, rewriting, SFT pipelines)
- 📈 Scaling laws and data–model–compute trade-offs
- 🏗️ Pretraining objectives & training dynamics (causal LM, masked LM)
- 🎯 Fine-tuning strategies (SFT, instruction tuning)
- 🤖 Model alignment (RLHF, DPO, RL algorithms)
- ⚡ Inference systems (latency vs throughput, batching, serving)
- 📏 Evaluation methodologies (perplexity, downstream benchmarks)
- 💻 GPU & TPU architecture + memory optimization
- 📉 Resource accounting (roofline), profiling & performance tuning
- 🌐 Distributed training (data/model/pipeline parallelism)
- 🔗 Collective communication (AllReduce, NCCL)
- 🧩 Mixture of Experts (MoE) and sparse architectures
- 🧬 Kernel-level optimization (CUDA, Triton)
- 🧠 Compiler stacks & graph optimization (XLA)
- 🚀 KV Cache, Flash Attention, efficient attention mechanisms
- 🔢 Mixed precision training (FP16, BF16)
- 🛠️ Checkpointing, fault tolerance, training efficiency
- 📦 System-level optimization (throughput, latency, memory–compute balance)
- 🌍 Deployment & LLM serving systems
- 🧠 RL policy optimization and RL aligment
- ➕ and more…
- 📄 Stanford_CS336_Handwritten_Notes.pdf — Complete handwritten notes (160+ pages)
This resource is intended for:
- Students studying Large Language Models
- AI and LLM engineers
- ML Systems engineers
- Researchers and scientists interested in ML System and AI infrastructure
- Anyone preparing to study or review Stanford CS336
Please note that these are personal study notes, not official course materials. They reflect my own understanding of the lectures and may contain omissions or mistakes.
Many thanks to Professor Percy Liang, Professor Tatsunori Hashimoto, the CS336 teaching staff, and everyone who contributed to making this outstanding course publicly available.
I am also grateful for the opportunity to contribute to the open-source CS336 course after taking it as a student.
These handwritten notes are shared for educational and non-commercial purposes. Please respect Stanford University's intellectual property and course policies. If you reuse or reference these notes, I would appreciate appropriate attribution.
⭐ If you find these notes helpful, please consider giving this repository a Star. I hope they make your journey into LLM engineering a little easier.