Deep learning from scratch

"What I cannot create, I do not understand." -- Richard Feynman

I agree.

---

Clean code implementation of the foundational deep learning layers, optimizers and models

• using PyTorch's autograd for the backpropagation
• using PyTorch's tensors for GPU computation

---

Building blocks

lib.layers	Linear Embedding BatchNorm 1 BatchNorm1d BatchNorm2d LayerNorm 2 RMSNorm 3 LocalResponseNorm 4 Dropout 5 RNN_cell LSTM_cell 6 GRU_cell RNN Conv2d ConvTranspose2d	Conv2dGroups Pool2d MaxPool2d AvgPool2d BatchAddPool SEGate 7 Graph_cell GCN_cell 8 GraphSAGE_cell 9 DiffPool 10 ReLU GELU 11 GLU 12 SwiGLU 13	Flatten DotProductAttention AdditiveAttention 14 DiagBlockAttention 15 ColumnBlockAttention 15 MultiHeadAttention 16 GroupedQueryAttention 17 RelativeWindowAttention SparseMultiHeadAttention 15 KVCache PositionalEncoding 16 RotaryEncoding 18 PatchEmbedding 19 RelativePositionBias2d 20
lib.autoencoders	MatrixFactorization	AutoencoderLinear	Word2Vec 21
lib.optimizers	Optimizer SGD SGD_Momentum AdaGrad	RMSProp AdaDelta Adam AdamW 22	LR_Scheduler LR_StepScheduler LR_PlateauScheduler LR_CosineDecayScheduler
lib.regularizers	L2_regularizer L1_regularizer	elastic_regularizer max_norm_constraint_	grad_clip_ grad_clip_norm_

Models / Networks

Family	Models
models.shallow_models	Perceptron, SVM, LeastSquareRegression, LogisticRegression, MulticlassPerceptron, MulticlassSVM, MultinomialLogisticRegression
models.energy_based_models	HopfieldNetwork, HopfieldNetworkOptimized, RestrictedBoltzmannMachine
models.recurrent_networks	RNN_factory, SimpleRNN, LSTM 6, GRU, LangModel, EchoStateNetwork, Encoder, Decoder, Seq2Seq 23
models.convolutional_networks	SimpleCNN, SimpleFullyCNN, LeNet5 24, AlexNet 4, NetworkInNetwork 25, VGG16 26, GoogLeNet 27, DeepPlainCNN
models.residual_networks	ResNet34 28, ResNet50 28, ResNeXt50 29, SEResNet50 30, SEResNeXt50 30, DenseNet121 31, UNet_DDPM
models.blocks.convolutional_blocks	Inception 27, ResBlock 28, ResBottleneckBlock 28, ResNeXtBlock 29, DenseLayer 31, DenseBlock 31, DenseTransition 31
models.graph_networks	GCN 8, GraphSAGE 9, GIN 32, DiffPoolNet 10
models.attention_networks	RecurrentAttention 33, SpatialTransformer 34, SpatialTransformerNet, AttentionEncoder 14, AttentionDecoder 14, BahdanauAttention 14
models.transformer_networks	TransformerEncoderLayer, TransformerEncoder, TransformerDecoderLayer, TransformerDecoder, Transformer 16, GPT_TransformerBlock 35, GPT_SparseTransformerBlock 15, GPT2 35, GPT3 36, LLaMA_TransformerBlock 37, LLaMA1 37, LLaMA2 38, LLaMA3 39
models.visual_transformers	VisionTransformer 19, VisionTransformerConvStem 40, SwinTransformerBlock 20, SwinTransformer 20
models.diffusion_models	DenoiseDiffusion 41

Example usages & experiments

examples/shallow	Binary classification Multi-class classification Matrix factorization MNIST classification Visualize optimizers
examples/energy_based	Memorize patterns with Hopfield Memorize more - optimized Hopfield Memorize with RBM
examples/recurrent	Predict next character Predict masked words Translate English to French word2vec embeddings
examples/convolutional	Image classification Feature maps visualization Saliency maps visualization
examples/graph	Classify nodes in a graph Classify protein graphs
examples/attention	Recurrent visual attention Translate with additive attention
examples/transformers	Translate with Transformer Predict next token (byte-pair) Reproduce GPT-2 performance
examples/diffusion	Generate colored digits

---

References

1. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
2. Layer Normalization
3. Root Mean Square Layer Normalization
4. ImageNet Classification with Deep Convolutional Neural Networks
5. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
6. Generating Sequences With Recurrent Neural Networks
7. Squeeze-and-Excitation Gate layer
8. Semi-Supervised Classification with Graph Convolutional Networks
9. Inductive Representation Learning on Large Graphs
10. Hierarchical Graph Representation Learning with Differentiable Pooling
11. Gaussian Error Linear Units (GELUs)
12. Language Modeling with Gated Convolutional Networks
13. GLU Variants Improve Transformer
14. Neural Machine Translation by Jointly Learning to Align and Translate
15. Generating Long Sequences with Sparse Transformers
16. Attention Is All You Need
17. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
18. RoFormer: Enhanced Transformer with Rotary Position Embedding
19. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
20. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
21. Efficient Estimation of Word Representations in Vector Space
22. Decoupled Weight Decay Regularization
23. Sequence to Sequence Learning with Neural Networks
24. Gradient-based learning applied to document recognition
25. Network In Network
26. Very Deep Convolutional Networks for Large-Scale Image Recognition
27. Going deeper with convolutions
28. Deep Residual Learning for Image Recognition
29. Aggregated Residual Transformations for Deep Neural Networks
30. Squeeze-and-Excitation Networks
31. Densely Connected Convolutional Networks
32. How Powerful are Graph Neural Networks?
33. Recurrent Models of Visual Attention
34. Spatial Transformer Networks
35. Language Models are Unsupervised Multitask Learners
36. Language Models are Few-Shot Learners
37. LLaMA: Open and Efficient Foundation Language Models
38. Llama 2: Open Foundation and Fine-Tuned Chat Models
39. The Llama 3 Herd of Models
40. Early Convolutions Help Transformers See Better
41. Denoising Diffusion Probabilistic Models

Installation

Local Setup

uv sync

Run examples

# Load the env
source .venv/Scripts/activate

# List all examples
find examples/ -name "*.py" -printf "python -m %p\n" | sed 's|/|.|g; s|\.py$||'

# Run an example:
python -m examples.attention.affine_mnist_spatial_transformer

Run tests

MPLBACKEND=Agg  pytest ./test