GitHub has released findings about Copilot Chat's impact on authoring and reviewing code. It reveals that the quality of code authored and reviewed with help from Copilot  Chat was higher than without, solidifying GitHub’s vision that embedding AI in the development workflows provides a better developer experience.

GitHub Copilot Chat was announced at Microsoft Build in May 2023 as part of Copilot X which will ring with it other AI-assisted features such as help with pull requests and automated test generation.

Copilot Chat, currently in beta, answers exclusively coding-related questions that you ask, by typing or voice, in natural language and returns them within the Visual Studio or Visual Code IDE, saving you from searching Google or Stack Overfow. While in beta, Copilot Chat is available free to those with a subscription to Copilot, something that costs $10 USD per month/ $100 USD per year to individuals or $19 USD per user per month.

As we reported at the time in GitHub Copilot Provides Productivity Boost, last year, following a survey of more than 2,000 developers, 88% of respondents reported they were more productive with Copilot than without it and an empirical study involving 95 developers assigned to two groups to complete a task with and without Copilot, revealed those using it were able to complete a task 55% faster.

Now another empirical study has looked at the impact of Copilot Chat on code quality and Mario Rodriguez has presented its results on the Github blog.

Github recruited 36 participants with between five and 10 years of software development experience and with some experience using Copilot but no experience of Copilot Chat. The participants  were randomly assigned to either use Copilot Chat or not and before using it were shown a brief video on its functionality.

All participants were asked to author API endpoints for an HTTP service that creates, reads, and deletes objects, making one pull request for their work on the create API endpoint, and another for the read and delete portion. Next developers were assigned the two pull requests that another participant in the study authored without knowing whether it was authored with or without Copilot. Finally the participants that originally authored the code reviewed the comments on their pull requests to decide which were helpful in improving the quality of the code and how actionable the comments were. Again, these participants were blind to which pull had been reviewed with Copilot Chat and which hadn't.

Participant rated the process of conducting the review with or without GitHub Copilot Chat. In addition they rated the code in terms of being error free and the code quality terms of five metrics readable, reusable, concise, maintainable and resilient. They also rated the comments in terms of being actionable: 

In each case there is a clear advantage in using Copilot Chat.

The chart below shows an overwhelming positive response to using Copilot Chat. Remembering that the survey had only 36 participants and they either used Copilot Chat or did not use it, I guess that the shortest bar corresponds to 1 person and that there was a single individual who strongly disagreed about being less frustrated/more focused/enjoying coding and being more confident. 

Putting together agree and strongly agree almost 88% of those using Copilot Chat spent less time on repetitive tasks and less time searching which made them less frustrated and more focused or more able to maintain flow state as report by GitHub in its key takeaways: 

Comparing the results with the previous study of the impact of Copilot as a pair programmer, the satisfaction rate of 88% is maintained while there isn't as dramatic a speed increase.

As Rodriguez comments:

But working fast is just one part of the picture—in many cases, there has traditionally been a tradeoff between doing something quickly and doing something right. As artificial intelligence continues to write code for an increasing number of developers, ensuring good code quality is even more important.

And this study shows that using Copilot Chat in authoring and reviewing code does indeed improve code quality. In terms of developer experience it improves developer confidence and enhances their enjoyment. A win-win scenario.

Overall this serves as a good example of generative AI assisting developers in a manner that enhances their experience and doesn't threaten to takeover their roles.

More Information

Quantifying GitHub Copilot’s impact on code quality

Related Articles

The Impact of Copilot Chat

Copilot X Star Of Show on Build Day 1

GitHub Copilot Provides Productivity Boost

GitHub Copilot Released

GitHub Copilot - Your Programming Pal

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info