Many professional developers find that coding assistants save them time and increase their productivity. But what about letting learners loose with these tools. Does it really help them?

Photo Credit: Communications of ACM

This thorny question is one that Eric Klopfer,  a professor and director of the MIT Scheller Teacher Education Program and The Education Arcade put to the test using two gen AI tools trained on large language models.

According to Esther Shein, writing in Communications of the ACM, Klopfer decided to conduct an experiment in his undergraduate computer science class at the Massachusetts Institute of Technology (MIT). He divided the class into three groups and gave them a programming task to solve in the Fortran language, which none of them knew. One group was allowed to use ChatGPT to solve the problem, the second group was told to use Meta’s Code Llama to help them out and the third group could only use Google. 

The group that used ChatGPT, and could simply formulate a straightforward prompt to be supplied with appropriate code solved the problem quickest. It took the second group, using Code Llama longer to solve it because they had to do rather more and well less spoon-fed, and it took the group using Google even longer, because they had to break the task down into components.

However using ChatGTP to essentially do your homework turns out not to be a long-term investment. When the the students were tested on how they solved the problem from the ChatGPT group “remembered nothing, and they all failed.” Meanwhile, half of the Code Llama group passed the test and so did all the group that used Google.

According to Klopfer:

“This is an important educational lesson. Working hard and struggling is actually an important way of learning. When you’re given an answer, you’re not struggling and you’re not learning. And when you get more of a complex problem, it’s tedious to go back to the beginning of a large language model and troubleshoot it and integrate it."

He also explains that breaking the problem into components allows you to use an LLM to work on small aspects, as opposed to trying to use the model for an entire project, concluding:

“These skills, of how to break down the problem, are critical to learn.”

This example has led Klopfer to confidently conclude that  computer science has not reached the end of the line but is in fact more important. This is the same conclusion reached by the Computer Science Teachers Association and Teach AI as I reported in Kids Still Need Computer Science Education.

In general experts agree that the more AI is used and the nature of jobs change, workers in almost every domain will need to be taught basic the concepts of programming together with an understanding of how the systems they are involved with work. Generative AI will be able to do the spade work, allowing humans to focus on the interesting and creative aspects.

More Information

The Impact of AI on Computer Science