The pain points of building a copilot - Austin Z. Henley

Interesting to read the approach and challenges teams are facing building "copilot" functions for their applications.

Given the nondeterministic nature of LLMs, it changes the typical software development process. We modeled the high-level process that the developers we interviewed are following: exploration, implementation, evaluation, and productization. It is not linear though, rather it is messy and iterative.

For example, there are additional risks that product teams are taking on. They must identify business scenarios that they think are feasible to solve using LLMs, tinker with various technologies to see if it works (e.g., LLMs, embeddings, vector databases), evaluate if it is good enough, and then they have to ship it to actual users.

Each and every one of these tasks has pain points for developers.

Reading this makes me feel pretty strongly that we will soon have frameworks or "kits" to do this and reduce or remove some of these challenges.