Host
intoli.com
1 link, saved on September 20, 2017.
- Analyzing One Million robots.txt Files
This is a delightfully fun read analyzing the robots.txt files of the top million websites. Interesting history about the origins of the robots specification and the fact that it…