Scaling Laws, Carefully

This article provides a comprehensive examination of scaling laws in deep learning, a critical empirical finding that describes how training loss decreases predictably with increases in model size, dataset size, and compute. It covers the mathematical formulation of scaling laws, including the approximation C≈6ND for training compute, and traces the historical development of loss predictability from early theoretical work by Amari et al. (1992) to empirical studies by Hestness et al. (2017). The article explains different types of learning curves based on data noise and algorithm determinism, and discusses practical applications such as fitting scaling laws on small runs to extrapolate requirements for larger models. It is a technical resource for researchers and engineers working in deep learning, model optimization, and AI infrastructure.