# Bots, Spiders, and Crawlers: The Results – Manu

<https://manuelmoreale.com/bots-spiders-and-crawlers-the-results>

- Host: [manuelmoreale.com](https://links.thingelstad.xyz/hosts/manuelmoreale.com/)
- Saved: September 28, 2023 (2023-09-28T12:53:55Z)
- Record: https://links.thingelstad.xyz/links/153e1a3e02fe/
- JSON: https://links.thingelstad.xyz/links/153e1a3e02fe.json

## What Jamie wrote

I've long assume robots.txt is ignored by most robots. Moreale's experiment is good proof of that. If you really want to blog robots, you'll need to do it on the HTTP server and it is going to be a big pain.

## Facts

- Kind: [Page](https://links.thingelstad.xyz/kinds/page/)

## Appears in

- [WT263: Weekly Thing 263 / Copilot, Del.icio.us, Pancake](https://weekly.thingelstad.com/archive/263/), Briefly

## Same host

- [Places on the web – Manu](https://links.thingelstad.xyz/links/661ee39b7484/) (manuelmoreale.com, September 4, 2023)
- [On Ad Blockers – Manu](https://links.thingelstad.xyz/links/2f42dc24ca48/) (manuelmoreale.com, November 21, 2023)
- [On POSSE – Manu](https://links.thingelstad.xyz/links/7a87e8d1a417/) (manuelmoreale.com, March 10, 2024)
- [The web is not dying – Manu](https://links.thingelstad.xyz/links/92c55991626e/) (manuelmoreale.com, April 28, 2024)
- [On hobbies, side projects, and money – Manu](https://links.thingelstad.xyz/links/237ba9b63f74/) (manuelmoreale.com, August 18, 2024)

## Saved the same week

- [The Screens are the Symptom | danah boyd](https://links.thingelstad.xyz/links/dcdff045f9ac/) (zephoria.org, September 28, 2023)
- [macOS Sonoma: The MacStories Review - MacStories](https://links.thingelstad.xyz/links/a404cc0abfad/) (macstories.net, September 28, 2023)
- [How Mastercard sells its ‘gold mine’ of transaction data](https://links.thingelstad.xyz/links/af210437ecf5/) (pirg.org, September 28, 2023)
- [The World’s Best Pancake Recipe](https://links.thingelstad.xyz/links/ee67517f3bb6/) (kottke.org, September 28, 2023)
- [Using ChatGPT as an editor – recursion by Luke Francl](https://links.thingelstad.xyz/links/9b485f67ec25/) (recursion.org, September 28, 2023)

## Elsewhere

- Wayback Machine: https://web.archive.org/web/*/https://manuelmoreale.com/bots-spiders-and-crawlers-the-results