The Effects of Generative AI on High Skilled Work

I’m not sure I would put too much weight into this study due to the small sample set of just three organizations (Microsoft, Accenture, Anonymous Company). It is good to see this studied and interesting to see the results.

Our preferred estimates suggest that usage of the coding assistant causes a 26.08% (SE: 10.3%) increase in the weekly number of completed tasks. When we look at outcomes of secondary interest, our results support this interpretation, with a 13.55% (SE: 10.0%) increase in the number of code updates (commits) and a 38.38% (SE: 12.55%) increase in the number of times code was compiled. For Microsoft we observe both the developers' tenure and their seniority as measured by job title. We find that Copilot significantly raises task completion for more recent hires and those in more junior positions but not for developers with longer tenure and in more senior positions. Prior work has shown that when workers are conducting the same tasks, generative AI helps lower-ability or lower- experience workers more (e.g., Brynjolfsson, Li, and Raymond 2023; Noy and Zhang 2023). Our results extend this finding by showing that even when workers are performing tasks according to their tenure or seniority, generative AI increases productivity more for lower-ability workers.

I feel like we need more understanding about the decrease benefit associated with seniority. One conclusion could be that they are more resistant to usage, or more habitual in their approach. It could also suggest that they find the LLM recommendations less effective?

However you approach it though it is clear that there is a benefit to be had for developers to use AI capabilities.