Simon Willison • 9/7/2026

Creepy crawlies

Konstantin Ryabitsev highlights the severe impact of abusive web crawlers on git.kernel.org, the official Git repository for the Linux kernel. The 'background radiation' of scrapers forces the system to spend more CPU cycles rendering commits for scrapers than on all other legitimate access, including git clones. Across five geo-distributed nodes, 14 CPU cores are constantly dedicated to rendering git commits as HTML for these crawlers. The author relates this to concerns for Datasette, which serves many crawlable pages, emphasizing the broader issue of crawler abuse affecting tech infrastructure.

0 comments

#git #web-crawlers #linux kernel