Notes on how to use LLMs in your product. | Irrational Exuberance
lethain.com/mental-model-for-how-to-use-llms-in-products
What Jamie wrote
Will Larson shares his "notes" as he has been working on adding LLM capabilities to products. There are some really good thoughts here that should help product people figure out how to approach this and result in better outcomes.
Some highlights I pulled out…
My belief is that many existing products will find they can only significantly benefit their user experience from LLMs by rethinking their workflows.
This makes a lot of sense, and the two phase combination of an unsophisticated algorithm to get plausible components of a response along with an LLM to filter through and package the plausible responses into an actual response works pretty well.
Because you cannot rely on LLMs to provide correct responses, and you cannot generate a confidence score for any given response, you have to either accept potential inaccuracies (which makes sense in many cases, humans are wrong sometimes too) or keep a Human-in-the-Loop (HITL) to validate the response.
Every large-scale adoption of LLMs today is being done in a mode where it shifts responsibility for the responses to a participating human.
Not a single line in this that I would disagree with. Great stuff.