Tracking exactly what happens to the internal "circuitry" (induction heads) of a 2-layer attention-only Transformer when forced to undergo domain adaptation from prose to structured Python code.

Mechanistic interpretability: reproducing + extending Olsson '22 induction heads on Pythia, with a Triton kernel for the hot probing path.

Does Quantization Kill Interpretability? Scaling study across 5 models (124M-2.8B): RTN destroys induction heads in small models, GPTQ preserves them at all scales.

Pre-registered toy-model study of how positional encoding (RoPE / learned / NoPE) shapes induction-circuit formation and ablation robustness in a 2-layer transformer. Predictions committed before the run; misses scored.

Reverse-engineering neural network internals from scratch in NumPy + PyTorch. A 6-week masterclass: linear representation hypothesis, superposition, sparse autoencoders, transformer circuits & induction heads, activation/path patching & causal scrubbing, and steering a real LM. Fully executed notebooks.

Reproduction of the induction-heads circuit (Olsson et al., 2022) in a 2-layer attention-only transformer trained on synthetic random-boundary repeat sequences, with mechanistic identification by ablation and training-time tracking of circuit formation.

Mechanistic interpretability of Zamba2-1.2B hybrid SSM — SSMI, copy scores, logit-lens factual recall. First analysis of weight-tied attention in hybrid architectures.