Explores recent LLM architecture innovations like KV sharing, compressed attention, and mHC for long-context efficiency.
Analysis of recent LLM architecture innovations including KV sharing, compressed attention, and mHC for reducing long-context costs.
A talk on implementing LLM architectures from scratch in Python and PyTorch, covering workflow for analyzing new models.
A talk on implementing LLM architectures from scratch in Python and PyTorch, focusing on tracing design changes in open-weight models.
A workflow for understanding open-weight LLM architectures using config files and code from Hugging Face.
A workflow for understanding new open-weight LLM architectures using config files and code from Hugging Face.
A visual guide to attention variants in modern LLMs, covering MHA, GQA, MLA, sparse attention, and hybrid architectures.
An overview of alternative LLM architectures beyond standard transformers, including linear attention hybrids, text diffusion models, and code world models.
An overview of alternative LLM architectures beyond standard transformers, including linear attention hybrids, text diffusion models, and world models.
