After early indications that AI-powered coding assistants increase developer productivity, findings are emerging that challenge this and suggest code quality may be reduced by their use. At the same time, a majority of developers now feel their jobs are threatened by AI.

Up until now we've reported positive findings about the impact of GitHub Copilot, typified by this chart from GitHub's AI In Software Development Survey 2024, which compared how AI tools were being adopted in the USA, the UK, India, Brazil and Germany,

As we previously reported, while there were regional variations, in general the findings confirmed that AI helps them work more productively, allowing them to focus on designing systems, collaborating more effectively, and better meeting customer requirements. 

Another recent report that reflected similar findings is from a survey 1,200 executives and IT professionals across the United States and United Kingdom conducted for Pluralsight by Wakefield Research to understand leaders’ and technologists’ perspectives on AI. This chart shows that Increased productivity and efficiency is a key driver of investing in AI

Now we have some research that challenges this perception.
A report from Uplevel, which describes itself as an Engineering Intelligence For Enterprise company found no significant gains in productivity through using GitHub Copilot. Using data generated by its customers, Uplevel the study compared two groups, 351 in a TEST group which had access to Copilot and 434 in a CONTROL  group without access to Copilot. 

The study measured the pull request (PR) cycle time, that is the time to merge code into a repository during "before Copilot" and "after Copilot" three-month and found no significant improvements for developers using Copilot, concluding:

When comparing PR cycle time ... Copilot neither helped nor hurt the developers in the sample, and also did not
increase coding speed.

More worrying was the finding that teams with Copilot access had a 41% increase in bug rate, leading Uplevel to comment:

"This suggests that Copilot access may impact code quality."

This conflicts with the GitHub survey's generally upbeat findings on code quality, but there was wide regional variation:

Asked about how the use of AI tools affected code quality, 90% of respondents in the USA and  81% of those in India reported a perceived increase in code quality when using AI coding tools. In comparison, only 61% of respondents in Brazil and 60% in Germany saw an increase in code quality while a decrease was perceived by 19% and 15% in Brazil and Germany respectively.

Whereas our report on the 2024 Stack Overflow Developer Survey stated that "70% of professional developers do not perceive AI as a threat to their jobs", the Pluralsight survey conflicts with this stating:

"Nearly 3 in 4 IT practitioners worry the skills they use in their daily role will become obsolete very quickly because of AI tools. And 69% think they’re at least somewhat at risk of being replaced by AI."

A bigger question that that remain unanswered is, can AI program above the mean. Given it is trained on data that includes the best, but mostly consists of suboptimal code, can it rise above this and generate something better? If the answer is no, then AI poses no threat to the good programmer.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info