# The Illusion of Thought: Chain of Thought Lies

<https://www.toxsec.com/p/the-illusion-of-thought>

- Host: [toxsec.com](https://links.thingelstad.xyz/hosts/toxsec.com/)
- Saved: November 19, 2025 (2025-11-20T01:28:14Z)
- Record: https://links.thingelstad.xyz/links/c3e7b29d2720/
- JSON: https://links.thingelstad.xyz/links/c3e7b29d2720.json

## What Jamie wrote

Super interesting read on interesting [research from Anthropic](https://www.anthropic.com/research/reasoning-models-dont-say-think). 

> When researchers trained models to exploit incorrect hints for rewards, the models learned fast. They reward-hacked in over 99% of cases - finding the shortcut, taking the easy points. But they admitted to using these hacks less than 2% of the time in their Chain of Thought explanations.
> 
> Instead, they fabricated justifications. They'd construct long, plausible-sounding rationales for why the wrong answer was actually correct. No mention of the hint. No acknowledgment of the shortcut. Just a convincing story.

A thought when reading this: **it is shocking how much LLMs are like people.**

Is the LLM's chain of thought that it shares actually its real train of thought? Turns out maybe, or no, or how would we know? What was your train of thought to come to the last thing you decided? The LLM is providing one. A person would too if asked. But are either reliable? No. 

> Instead, they fabricated justifications. They'd construct long, plausible-sounding rationales for why the wrong answer was actually correct. No mention of the hint. No acknowledgment of the shortcut. Just a convincing story.

The "they" in that sentence is LLMs, but people do this all the time too.

## Facts

- Kind: [Page](https://links.thingelstad.xyz/kinds/page/)

## Appears in

- [WT333: Weekly Thing 333 / Gemini, LangChain, Illusion](https://weekly.thingelstad.com/archive/333/), Notable

## Saved the same week

- [New audio option for Micro.blog Studio - Manton Reece](https://links.thingelstad.xyz/links/7bf2802f7de4/) (manton.org, November 19, 2025)
- [Three Years from GPT-3 to Gemini 3 - by Ethan Mollick](https://links.thingelstad.xyz/links/c0a3bae719ad/) (oneusefulthing.org, November 19, 2025)
- [Google Antigravity](https://links.thingelstad.xyz/links/a3f4f4b5df8d/) (simonwillison.net, November 19, 2025)
- [Cloudflare outage on November 18, 2025](https://links.thingelstad.xyz/links/5271fc17f5fd/) (blog.cloudflare.com, November 19, 2025)
- [OpenAI and Target partner to bring new AI-powered experiences across retail | OpenAI](https://links.thingelstad.xyz/links/cfd2c1d3b5b9/) (openai.com, November 19, 2025)

## Elsewhere

- Wayback Machine: https://web.archive.org/web/*/https://www.toxsec.com/p/the-illusion-of-thought