# LLMs are weird, man – Surfing Complexity - Host: [surfingcomplexity.blog](https://links.thingelstad.xyz/hosts/surfingcomplexity.blog/) - Saved: May 29, 2025 (2025-05-30T03:33:39Z) - Record: https://links.thingelstad.xyz/links/789683250309/ - JSON: https://links.thingelstad.xyz/links/789683250309.json ## What Jamie wrote This is an enjoyable read highlighting how the inner workings of LLMs are a mystery to everyone, including those who create them. The point of this is that usually we can generate some sort of a mental model to categorize and understand a given technology, but LLMs seem to not fit those models. In a super simple way you can think of it like a "calculator for words" but that feels completely wrong when you see an LLM respond with something that is truly insightful. > But I think it's a mistake to write off this technology as _just_ a statistical model of text. I think the word "_just_" is doing too much heavy lifting in that sentence. Our intuitions break down when we encounter systems beyond the scales of everyday human life, and LLMs are an example of that. It's like saying "humans are _just_ a soup of organic chemistry" (c.f. Terry Bisson's short story [They're Made out of Meat](https://www.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html)). Intuitively, it doesn't seem possible that evolution by natural selection would lead to conscious beings. But, somehow we humans are an emergent property of long chains of amino acids recombining, randomly changing, reproducing, and being filtered out by nature. The scale of evolution is so unimaginably long that our intuition of what evolution can do breaks down: we probably wouldn't believe that such a thing was even possible if the evidence in support of it wasn't so damn overwhelming. It's worth noting here that one of the alternative approaches to AI was inspired by evolution by natural selection: _genetic algorithms_. However, this approach has proven much less effective than artificial neural networks. We've been playing with artificial neural networks on computers [since the 1950s](https://en.wikipedia.org/wiki/Perceptron), and once we scaled up those artificial neural networks with large enough training sets and a large enough set of parameters, and we hit upon effective architectures, we achieved qualitatively different results. In the late 90's before we created BigCharts and got a successful company up and running around market data visualization we had a very different product called The Conductor. That original product was Philip Hotchkiss first vision of informing trading models using neural nets and genetic algorithms. In modern terms the dozen or so computers we had (no GPUs!) using these advanced methods is almost comical. But the actions we were taking were advanced. Genetic algorithms would compete for superiority. Neural nets would predict future market behaviors. We graded them. We would find a "model" that did really well and promote it. Several weeks later that same model had become less effective and possibly overtrained itself so we would delete it. At that scale, dozens of cores of compute, not enough data to train on to care, it was all fairly simple but even at that level we had no idea "why" one model did better than the other. We only judged them on the outputs. But it turns out that if you scale the crap out of this it is different. Increase by dozens of orders of magnitude the data set, the compute, the storage — you get a very different thing on the other end. And that same challenge of not really knowing what is going on at the core is still there too. Why does this model work better than the other? Shrug. I’m far out of my skis on this topic. It is notable to me that extreme scale can result in just something very different. Don't brains have the same thing in the extreme? Increase the number of neurons by several orders of magnitude and at some point you get different outcomes? We don't really know, but I agree with the author that this stuff is "weird", and it is also rather amazing. ## Facts - Kind: [Page](https://links.thingelstad.xyz/kinds/page/) ## Appears in - [WT321: Weekly Thing 321 / Saluting, Tetris, Sky](https://weekly.thingelstad.com/archive/321/), Notable ## Same host - [A small mistake does not a complex systems failure make – Surfing Complexity](https://links.thingelstad.xyz/links/5db9484f5814/) (surfingcomplexity.blog, January 23, 2023) - [Brief thoughts on the recent Cloudflare outage – Surfing Complexity](https://links.thingelstad.xyz/links/c64bc55e1cbf/) (surfingcomplexity.blog, December 1, 2025) ## Saved the same week - [Lazy Tetris](https://links.thingelstad.xyz/links/478e47db51ef/) (lazytetris.com, May 29, 2025) - [TIL: timeout in Bash scripts | Heitor's log](https://links.thingelstad.xyz/links/1be2503d8f8b/) (heitorpb.github.io, May 29, 2025) - [CSS Minecraft](https://links.thingelstad.xyz/links/3c7bf8779cd5/) (benjaminaster.com, May 29, 2025) - [I made a font | blog.chay.dev](https://links.thingelstad.xyz/links/a6c69db1b643/) (blog.chay.dev, May 29, 2025) - [Russia launches one of biggest drone attacks on Kyiv since start of war | Ukraine | The Guardian](https://links.thingelstad.xyz/links/8338e540ac26/) (theguardian.com, May 29, 2025) ## Elsewhere - Wayback Machine: https://web.archive.org/web/*/https://surfingcomplexity.blog/2025/05/25/llms-are-weird-man/