Findings from GitHub show that code authored with Copilot has increased functionality and improved readability, is of better quality, and receives higher approval rates than code authored without it.

A year ago research from GitHub revealed that 85% of developers felt more confident in their code and 88% felt more in the flow when using GitHub Copilot, see Copilot Chat Improves Confidence and Enjoyment

More recently findings from Statista.com suggested that 44% of developers use GitHub Copilot on a regular basis. This level of adoption suggests we need to look into the question of whether the quality of code written using Copilot is better or worse that without it.

To address this, the GitHub Customer Research team conducted a randomized controlled trial to understand how functional, readable, reliable, maintainable, concise, and likely to be approved code authored using GitHub Copilot is. 

For the study, the GitHub researchers recruited 202 developers with at least five years of Python experience. Half were randomly assigned GitHub Copilot access and the other half were instructed not to use any AI tools. The participants were all asked to complete the same coding task writing API endpoints for a web server and the code produced was evaluated using unit tests. In terms of functionality, the unit tests revealed that code authored with Copilot assistance was clearly better with 61% of developers' code passing all 10 unit tests compared to 39% among those not using Copilot.

The next part of the study was an expert review conducted by developers whose code had passed all 10 unit tests. They were randomly assigned to do a blind review of the anonymized submissions, both those written with and without GitHub Copilot. What the reviewers found was that code authored using GitHub Copilot had fewer code readability errors. Specifically, using GitHub Copilot wrote 18.2 lines of code per code error, but only 16.0 without.

In the blog post discussing these findings, Jared Bauer writes:

That equals 13.6% more lines of code with GitHub Copilot on average without a code error (p=0.002). This can translate into real-time savings, as each of these code errors require action from a developer. For example, without using GitHub Copilot, teams may face up to 13% more comments or suggestions to address, which accumulates over time.

The reviewers also rated the code authored using GitHub Copilot as more readable by 3.62%; more reliable by 2.94%, more maintainable by 2.47%, and more concise by 4.16%. Although these differences were small, they were statistically significant and contributed to a better codebase.

As the final stage, reviewer looked at whether code authored with assistance from Copilot was more likely to be approved than code authored without such assistance and found that it was:

The expert reviewers were 5% more likely to approve code authored using GitHub Copilot, leading Bauer to comment:

In real-world settings, this means that developers using GitHub Copilot write code that’s ready to be merged sooner, which will speed up the time to fix bugs or deploy new features.

Expanding on the findings of the research study, Bauer writes:

While the number of commits and lines of code changed was significantly higher for the GitHub Copilot group, the average commit size was slightly smaller. This suggests that GitHub Copilot enabled developers to iterate on the code to improve its quality.

The research team's conclusion was that because developers spent less time making their code functional, they were able to focus more on refining its quality. This finding aligned with its previous findings that developers felt more confident using Copilot. 

More Information

Does GitHub Copilot improve code quality?

Related Articles

Copilot Chat Improves Confidence and Enjoyment

GitHub Copilot Provides Productivity Boost

The Impact of Copilot Chat

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info