Dan Luu • 7/23/2026

Exercises in benchmarking and evals, part 7: DeepSWE, Senior SWE-Bench, napkin math, and winter tires

This article is part of a series on benchmarking, evals, and experimental design. It examines three questions: first, a critique of napkin math performance estimates used for computer performance interviews, pointing out issues with the benchmark table; second, an analysis of DeepSWE and Senior SWE-Bench benchmarks used to compare AI models, questioning their validity; third, a discussion of winter tire superiority in cold weather as an analogy for benchmarking fallacies. The content is focused on technical benchmarking, performance evaluation, and critical thinking in computer science contexts.

0 comments

#latency #benchmarking #Eval