DeepSeek Sparse Attention implementation from scratch, featuring a learned sparse pattern replacing fixed attention with a lightweight indexer and selector.
Explores recent LLM architecture innovations like KV sharing, compressed attention, and mHC for long-context efficiency.
Analysis of recent LLM architecture innovations including KV sharing, compressed attention, and mHC for reducing long-context costs.
A detailed comparison of architectural developments in major large language models (LLMs) released in 2024-2025, focusing on structural changes beyond benchmarks.
A technical comparison of architectural changes in major Large Language Models (LLMs) from 2024-2025, focusing on structural innovations beyond benchmarks.
An updated, comprehensive overview of the Transformer architecture and its many recent improvements, including detailed notation and attention mechanisms.
