Is there a Half-Life for the Success Rates of AI Agents? — Toby Ord
www.tobyord.com/writing/half-life
What Jamie wrote
Frankly I’m amazed at the accuracy for these tasks at these durations. I figured it would be a 1 or 5 minute task, not 59 minutes.
For the best model (Claude 3.7 Sonnet) it could achieve a 50% success rate on tasks up to 59 minutes vs only 15 minutes if an 80% success rate was required. If those results generalise to the other models, then we could also see it like this: the task length for an 80% success rate is 1/4 the task length for a 50% success rate. Or in terms of improvement: what is doable with a 50% success rate now is doable with an 80% success rate in 14 months' time (= 2 doubling times).
Also the rate of improvement is fast.