James Turner (Turnerj) • 4/13/2019

Building a Polite Web Crawler

This technical article discusses the importance of building a 'polite' web crawler that minimizes server load and respects website rules. It covers best practices like adhering to robots.txt, using a custom User Agent, and managing crawl frequency. The author also introduces 'InfinityCrawler,' a .NET Standard library they created to meet these needs with async/await and .NET Core support.

0 comments

#http requests #robots.txt #web scraping