The 39-point gap: why developers can't tell if AI is making them faster
The experiment: measuring AI's real effect on developers
There is a study everyone deploying AI in engineering should read: in July 2025 the research nonprofit METR ran a randomised controlled trial on whether AI tools actually make developers faster.
Sixteen experienced open-source developers, working on codebases they had maintained for years, took 246 real tasks from their own projects. Each task was randomly assigned to allow AI or forbid it, mostly Cursor with Claude 3.5 and 3.7 Sonnet, the best models available that spring. The developers predicted AI would make them 24% faster. Afterwards they said it had made them about 20% faster. Measured, the tasks done with AI took 19% longer.
The forty-point gap between feeling and reality
The slowdown is the catchy headline, but it is probably not the most interesting part. The interesting part is that these developers were sure it had gone the other way. They lived through the tasks, did the work with their own hands, and came out believing AI had sped them up by 20% when it had slowed them down by 19%. That is a gap of nearly forty points between what happened and what they thought happened. And these were not people new to the tools. They knew the models, they knew the code, they were being paid to pay attention. But they still could not tell.
This is very relevant for how many companies decide their AI strategy: the tooling budget, the rollout plan, the board slide about transformed productivity… Nearly all of it rests on people saying how much faster they feel. METR took that exact measure and showed it can be wrong by forty points in expert hands under controlled conditions. A survey asking your engineers whether AI is helping is not measuring output. It is measuring enthusiasm.
Why the follow-up study almost didn't happen
The obvious objection is that the tools have moved on, and yes, they have. So the more useful part of the story is what happened when METR tried to run the study again. They started a second version, more developers, newer tools, a wider set of projects.
And it did not really work.
Developers stopped cooperating. A growing share turned the study down once they understood that half their tasks would have to be done without AI, even at $50 an hour to work on their own code. Of those who did take part, between a third and a half admitted they were quietly leaving out certain tasks, the ones where AI would have saved them the most, because they could not face doing that particular work the old way. One described it as trying to walk for miles across the city after getting used to taking an Uber.
So the experiment that ran cleanly in early 2025 was close to unrunnable, because the thing it was measuring had become something people would no longer switch off. METR's own reading is that developers are probably faster with AI now, likely by a decent margin, but that the people dropping out and the tasks going unrecorded have bent the numbers too far to say by how much. Their raw figures lean towards a speedup, with confidence intervals wide enough to be nearly useless. This is the most careful team working on the question, and their honest answer is that AI probably helps and they cannot yet measure it. That answer is worth more than a precise one, because they had every chance to invent a number and did not.
What a CTO should actually measure
Where does this leave a CTO who has to actually decide and deliver things? Not doubting the tools, which plainly do real work, and not just trusting the dashboards either. It leaves you measuring what you can see instead of what people feel.
That means the delivery numbers, not the survey. How long a change takes from first commit to running in production. How often changes fail, and how long recovery takes, because more code and more incidents is not speed in any sense the business recognises. How much of the week goes on review rather than writing, which is where a lot of the AI-era cost has quietly moved. The DORA research has spent a decade showing these metrics track how organisations really perform, and they have the advantage of being hard to argue with. If a team is certain AI has transformed its velocity but the cycle time and failure rate have not moved, that is worth knowing. It is just not the thing the team thinks it knows.
The takeaway
None of this is an argument against agentic development. The tools are getting better quickly and the teams that adopt them with some judgement will be ahead of the ones that do not. The narrow point is this. The most common input into AI strategy today is how productive people say they feel, and that is the one input a proper trial has shown to be actively misleading, with even the careful researchers finding the ground moving under them faster than they can measure it. If your read on whether this is working comes mainly from what your teams tell you, the evidence says that is the one foundation that will not hold.