# Analyzing One Million robots.txt Files

<https://intoli.com/blog/analyzing-one-million-robots-txt-files/>

- Host: [intoli.com](https://links.thingelstad.xyz/hosts/intoli.com/)
- Saved: September 20, 2017 (2017-09-20T09:47:53Z)
- Record: https://links.thingelstad.xyz/links/66456674ff71/
- JSON: https://links.thingelstad.xyz/links/66456674ff71.json

## What Jamie wrote

This is a delightfully fun read analyzing the `robots.txt` files of the top million websites. Interesting history about the origins of the robots specification and the fact that it was never standardized. There isn't even an RFC for it! As a bonus the code excerpts to show the analysis are all in Python and was the first time I've seen the [collections](https://docs.python.org/2/library/collections.html#module-collections) module.

## Facts

- Kind: [Page](https://links.thingelstad.xyz/kinds/page/)

## Appears in

- [WT20: Weekly Thing for September 23, 2017](https://weekly.thingelstad.com/archive/20/), Links 📌

## Saved the same week

- [Swift.org - Swift 4.0 Released!](https://links.thingelstad.xyz/links/2aa8c47b1677/) (swift.org, September 20, 2017)
- [Satya Nadella Rewrites Microsoft’s Code](https://links.thingelstad.xyz/links/afd8efa0c6dd/) (fastcompany.com, September 20, 2017)
- [How Useful is Tufte for Making Maps? | Making Maps: DIY Cartography](https://links.thingelstad.xyz/links/14679c3e11e6/) (makingmaps.net, September 20, 2017)
- [Facebook’s war on free will | Technology | The Guardian](https://links.thingelstad.xyz/links/bb12d68df20b/) (theguardian.com, September 20, 2017)
- [★ Apple Watch Series 3 —Daring Fireball](https://links.thingelstad.xyz/links/7974a268c379/) (daringfireball.net, September 20, 2017)

## Elsewhere

- Wayback Machine: https://web.archive.org/web/*/https://intoli.com/blog/analyzing-one-million-robots-txt-files/