DeepSeek Sparse Attention From Scratch

This article details a from-scratch implementation of DeepSeek Sparse Attention added to the LLMs-from-scratch repository. It explains how the mechanism replaces fixed sparse patterns with learned ones using a lightweight indexer and selector to decide which prior tokens to attend to. The implementation includes a README, a standalone GPT-style reference implementation, and tests. The article also provides comparisons with regular causal attention and sliding-window attention, making it a valuable resource for developers and researchers interested in advanced attention mechanisms in large language models.