The LLMs-from-scratch GitHub repo hits 100k stars; author shares highlights and future plans for new architectures and projects.
Analysis of Kimi K3's open-weight architecture, focusing on LatentMoE, attention residuals, and NoPE innovations for inference efficiency.
DeepSeek Sparse Attention implementation from scratch, featuring a learned sparse pattern replacing fixed attention with a lightweight indexer and selector.
Explores recent LLM architecture innovations like KV sharing, compressed attention, and mHC for long-context efficiency.
Analysis of recent LLM architecture innovations including KV sharing, compressed attention, and mHC for reducing long-context costs.
A detailed comparison of architectural developments in major large language models (LLMs) released in 2024-2025, focusing on structural changes beyond benchmarks.
A technical comparison of architectural changes in major Large Language Models (LLMs) from 2024-2025, focusing on structural innovations beyond benchmarks.
An updated, comprehensive overview of the Transformer architecture and its many recent improvements, including detailed notation and attention mechanisms.
