Speaking of over-optimism, I had to chuckle when I read this article in the UK Daily Telegraph entitled, “The great AI delusion is falling apart: New research suggests the chorus of techno-optimism is based on falsehoods.” Clearly, it’s not the first (or the last) article to express scepticism about the benefits of AI, but the author cites an interesting METR research experiment.
“Some fascinating new research suggests that self-deception plays a key role in whether AI is perceived to be a success or a dud. In a randomised controlled trial – the first of its kind – experienced computer programmers could use AI tools to help them write code. What the trial revealed was a vast amount of self-deception.”
“The results surprised us,” research lab METR reported. “Developers thought they were 20pc faster with AI tools, but they were actually 19pc slower when they had access to AI than when they didn’t.”
“In reality, using AI made them less productive: they were wasting more time than they had gained. But what is so interesting is how they swore blind that the opposite was true. If you think AI is helping you in your job, perhaps it’s because you want to believe that it works.”
But it was the last paragraph that really made me chuckle.
“METR added an amusing footnote to their study. The researchers used one other control group in its productivity experiment, and this group made the worst, over-optimistic estimates of all. They were economists.”