GitHub set out to discover the impact its new AI-based tool, Copilot is having on developers' productivity. The results seem pretty conclusive - 88% of developers surveyed reported they were more productive and an empirical study revealed developers were 55% faster coding with Copilot than without it. 

Launched in technical preview in June 2021, GitHub Copilot is described as "Your AI pair programmer" which uses the OpenAI Codex to suggest code and entire functions in real-time from your editor.

As Mike James explained in our report GitHub Copilot - Your Programming Pal:

GitHub Copilot is an AI-based system that will take notice of what you are coding and make suggestions for what it is you are about to type. You can think of it as IntelliSense on steroids, but users are claiming that it is a big step up. The idea is all based on the sort of AI that lead to the GP-3 language system. Take a very big network and train it on a huge corpus of natural language data and it will extract the regularities.

More details were provided by Kay Ewbank who reported on Copilot's release in June 2022: 

The system has been trained to work out how to complete lines and sections of code by taking completed code, deleting parts of it, and training the AI to correctly fill in the missing part. The AI means that the examples used are remembered, along with the underlying principles so that the system is capable of extrapolation to things that it has never seen.

By the time of its release it worked with with Neovim, JetBrains IDEs, Visual Studio, and Visual Studio Code.

From the outset GitHub was confident that Copilot would improve developer productivity and now Eirini Kalliamvakou has revealed the research findings to prove it, both qualitative and quantitative, on the GitHub blog. 

The qualitative results - which rely on user's perceptions - are from a survey of more than 2,000 developers in which questions were based on the SPACE productivity framework. SPACE is an acronym for Satisfaction, Performance, Activity, Communication, and Efficiency and the framework, which was devised by Microsoft and GitHub in collaboration with University of Victoria had previously been used by GitHub for its Good Day Project, which I reported on in What Ruins A Developer's Day.  

With regards to Productivity, an overwhelming 88% of respondees agreed with the statement "I am more productive".

As to Satisfaction, 59% felt less frustrated when coding;  60% reported being more fulfilled with their job; and 74% agreed that using Copilot let them focus on more satisfying work.

Questions on Efficiency elicited the findings that Copilot helped them stay in the flow (73%) and preserve mental effort during repetitive tasks (87%). The researchers comment:

That’s developer happiness right there, since we know from previous research that context switches and interruptions can ruin a developer’s day, and that certain types of work are draining. 

Using Copilot also led to specific productivity gains with 88% reporting faster completion in general and 96% being faster with repetitive tasks.

To provided quantitative evidence the researchers conducted an empirical test in which 95 developers were split into two groups to determine their speed in completing the task of writing a web server in JavaScript with and without Copilot.

As shown in the infographic below those who used Copilot had a higher rate of completing the task (78%, compared to 70% for non-users) and even more impressive Copilot users were able to complete tasks 55% faster than those who didn’t use it.

At the end of the day the advantages of using AI Pair Programming seem to be pretty clear are are neatly summed up by a quote from a Senior Software Engineer who participated in the research:

“[With Copilot] I have to think less, and when I have to think it’s the fun stuff. It sets off a little spark that makes coding more fun *and* more efficient.”
 

More Information

Copilot Website

Related Articles

GitHub Copilot Released

GitHub Copilot - Your Programming Pal

What Ruins A Developer's Day   

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info