SAN FRANCISCO — A randomized controlled trial published today by the non-profit research group METR finds that experienced open-source developers who used early-2025 AI coding tools took 19% longer to complete real tasks than when working without AI, even though the same developers believed AI made them roughly 20% faster.

The study recruited 16 developers from large open-source repositories averaging 22,000+ stars and 1 million+ lines of code, and assigned them 246 real issues — bug fixes, features, and refactors — randomly split between AI-allowed and AI-forbidden conditions. When AI was allowed, developers primarily used Cursor Pro with Anthropic’s Claude 3.5 and 3.7 Sonnet, frontier models at the time. Developers self-reported completion times while being paid $150 per hour.

METR’s analysis rules out many experimental artifacts: developers used frontier models, complied with treatment assignment, submitted similar-quality pull requests in both conditions, and did not differentially drop hard issues. The researchers identify five likely contributing factors, including time spent prompting AI, waiting for responses, and AI struggles with large, complex codebases.

The result clashes sharply with developer self-assessments: participants forecasted a 24% speedup before the study, and even after experiencing the slowdown, still believed AI had sped them up by 20%. It also runs counter to benchmark scores on tests like SWE-Bench Verified and widespread anecdotal claims of AI productivity gains. METR explicitly says the study does not show AI never helps — only that in this particular setting with experienced developers on familiar codebases, the measured effect is negative.

The research group, which studies whether AI systems might threaten catastrophic harm, plans to repeat the methodology to track changes over time as AI tools evolve. The paper notes that learning effects from using tools like Cursor may only appear after hundreds of hours, and that AI capabilities may be lower in settings with very high quality standards. In the developer community, the finding is already fueling debate about whether benchmarks and anecdotal reports systematically overestimate real-world AI usefulness.