Building Etsy Buyer Profiles with LLMs | Etsy Engineering

It is interesting to me how LLM's are so different than Machine Learning models to architect solutions. Personally I've been able to "think different" about these approaches when I scale a large data problem down to a single instance and then think "How would a person do this for one thing?" In general that answer is a reasonable parallel to how an LLM could do it. Notably, it is very different than how a traditional Machine Learning and Big Data approach would work.

Of course the issue than is how you scale it, but there are many ways to approach that that they go over.

Initially, using this method to generate profiles for our roughly 90 million buyers would have been exceedingly costly and taken weeks to update. We optimized this by:

• Shifting our listings data source from API endpoints to BigQuery tables that are clustered and partitioned for efficient querying.
• Decreasing the volume of input tokens. Initially, we were including about 2 years of session data. Now, we've reduced this to just the last 9 months. The 9 month timeframe also allowed us to lighten the prompt corrections (and cached input tokens) by reducing the weight of holiday shopping.
• Increasing LLM and BigQuery batch sizes for data processing.
• Introducing parallel processing with managed concurrency to avoid request rate limits.
• Scaling up computational resources for session data retrieval and LLM processing tasks.

These improvements reduced buyer profile generation time dramatically -- from 21 days down to 3 days for 10 million users. Cost management was also crucial. By adjusting the prompt to get high quality results with a smaller model, we significantly lowered the cost and made large-scale personalization economically feasible. Through these various cost management techniques, we were able to reduce the estimated cost by 94% per million users.

The magical things I’m noting in these solutions is the ability to control the amount of data you bring to the LLM, the frequency, and even the cognitive effort the LLM will put towards it with a token budget.