Expanding on what we missed with sycophancy

Willison's commentary on OpenAI recent issue with "sycophancy" in the model. A recent model from OpenAI was overly praising — and for those that have used AI a lot you may be thinking they already do a pretty decent job of making you feel smarter and better than you probably are. This excerpt from Willison caught my eye.

Here's yet more evidence that the entire AI industry runs on "vibes":

In addition to formal evaluations, internal experts spend significant time interacting with each new model before launch. We informally call these "vibe checks"--a kind of human sanity check to catch issues that automated evals or A/B tests might miss.

This tickles a bit of unknown that bothers me in these LLM's and also bothered me with Neural Nets in general — they are black boxes to everyone. Nobody really knows what happens inside. So how do you train them — with "vibe checks"? Sounds a lot like how we dealt with way smaller models.

On one hand I can get comfortable with this. Neuroscientists don't specifically understand how our brains work after all, and we don't discount all input from our brain because of that. Mystery at the center of it all may be just fine.

However it does put in stark contrast a need for determinism. In computing we have many things that fundamentally need to be deterministic. We need to be mindful of the problem we are solving, and insure that an open approach, where we fundamentally don't know the "why" of any of it, is okay.