Making it easier for websites to express AI training preferences

openai.com/index/helping-websites-express-ai-training-preferences

I can tell you that on any given day about 10% of the traffic to all of my websites is from malicious actors, and then depending on the day about 30-40% is from crawlers. The web is full of robots these days. There are also a lot of folks, particularly in the IndieWeb space that are actively working to fight away bots and don't want anyone indexing their sites.

I'm going the complete other way. I have open robots.txt on all my sites. I've adopted the llms.txt standard for nearly every site I have. I even have many of my sites publishing static JSON objects to make it easier for agents and other code to interface with them. Check out the full LLMs.txt for my Things I've Built.

My plan is to lean into discoverability. Lean into what I think of as the true sharing nature of the web. Regardless of who or what is on the other side.

Weekly Thing
WT355 — Writing, Building, and the Open WebNotable, 6th of 28 links. Sent Saturday, November 14, 2026, 13 days after it was saved.All 28 links in WT355
Site
openai.comOpenAI | Research & DeploymentDomain registered January 2007
Kind
Page
Sky
OpenAI and the AI industryIts starand the ten links nearest it in meaning
Archive
32 Wayback Machine capturesOctober 2026 to November 2026. The first was two days before it was saved.
This record
JSON · Markdown