Lessons after a half-billion GPT tokens - Ken Kantzer's Blog
kenkantzer.com/lessons-after-a-half-billion-gpt-tokens
What Jamie wrote
Kantzer shares some key observations from his work building functionality with LLMs.
- When it comes to prompts, less is more.
- You don’t need langchain. You probably don’t even need anything else OpenAI has released in their API in the last year. Just chat API. That’s it.
- Improving the latency with streaming API and showing users variable-speed typed words is actually a big UX innovation with ChatGPT.
- GPT is really bad at producing the null hypothesis.
- “Context windows” are a misnomer – and they are only growing larger for input, not output.
- Vector databases, and RAG/embeddings are mostly useless for us mere mortals.
- Hallucination basically doesn’t happen.
He goes into details on each of these points. There is some good learnings here.