A whitepaper from GitClear sets out to investigate the quality and maintainability of AI-assisted code compared to what would have been written by a human. It highlights a worrying increase in "code churn" due to reliance on Copilot. 

We've previously reported on devs' enthusiasm for ChatGPT, Copilot and other forms of AI assistance and have reported on claims for improved productivity. Now some research has been published that raises doubts. It comes from GitClear and the findings are reported in a whitepaper:

Backing up the reference in the subtitle to the negative impact Copilot is having on code quality, the report shared a tweet from Adam Tornhill, the author of Your Code as a Crime Scene:

making the comment:

GitHub claims that code is written "55% faster" with Copilot. But what about code that shouldn't be written in the first place? The problem here is that code spends 10x more time being read than being written, according to Robert Martin, author of Clean Code: A Handbook of Agile Software Craftsmanship. Writing bad code faster implies considerable pains for the subsequent code readers.

To investigate the influence of AI-assistance GitClear used its Diff Delta tool to analyze 153 million changed lines of code, authored between January 2020 and December 2023 and projected the trends for 2024:

GitClear noted that since the introduction of Copilot in 2022 the percentage of 'added code' and 'copy/pasted code' had increased in proportion to 'updated,' 'deleted,' and 'moved 'code commenting:

In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited."

What GitClear finds most concerning is the increase in "churn" which it defines as the percentage of lines of code that are reverted or updated less than two weeks after being authored, showing that it is projected to double in 2024 compared to its 2021, pre-AI baseline:

The paper concludes:

There's no question that, as AI has surged in popularity, we have entered an era where code lines are being added faster than ever before. The better question for 2024: who's on the hook to clean up the mess afterward?" 

Hopefully the answer to this is developers aided by Copilot.
AI-powered coding tools are expected to learn from their own mistakes and as they see more and more lines of code they will increasingly distinguish between the good, bad and the ugly. Copilot will be able not only to identify the substandard code that it helped to generate, but also to refactor it.

After all, we are looking to AI to take on the spade work and that includes mucking out the stables as well as providing fresh fodder. 

More Information

Coding on Copilot

Related Articles

Copilot Chat Improves Confidence and Enjoyment

GitHub Copilot Provides Productivity Boost

The Impact of Copilot Chat

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info