How ChatGPT impacts Computer Science Education

Since its release in late 2022, ChatGPT has been the talk of the town. ChatGPT allows users to chat with an AI language model (”GPT-3.5” to be exact) in an easy-to-use environment for free. With just simple prompts, users can task ChatGPT to answer questions, explain complicated material or write full essays, papers, poems, songs, code or anything you can think of.

Computer Science teachers have reacted with concern to the sudden rise of ChatGPT. And understandably so: ChatGPT can easily pass the 2022 AP Computer Science A exam and early survey’s show already 30% of students in the US have used ChatGPT in their assignments. In this article, we will discuss the challenges that ChatGPT poses on current Computer Science education. We will also attempt to explore first steps you can take to make your coding course more resilient against ChatGPT, but most importantly discuss whether that is needed in the first place.

Want to learn more about ChatGPT in Coding Education? Join our live webinar on June 1st 2023, sign up below:

Power and limitations of ChatGPT for code

Before we dive into the consequences of ChatGPT on Computer Science education, let’s take a step back and discuss what it exactly is. ChatGPT is a large language model based on the GPT-family of algorithms, in short: it is trained to generate plausible text, as if written by a human.

What this means is that, even though the text or code ChatGPT produces is often surprisingly correct, it is not trained to generate true statements. This is why Princeton Computer Science professor Arvind Narayanan has coined the term ‘Bullshit Generator’. The output of ChatGPT may sound very convincing, but many have reported it generating made-up non-existing references and StackOverflow quickly banned ChatGPT because of the many incorrect answers it generated.

But, taking into account that the output is not necessarily true, our own tests can only conclude that for (easier) Computer Science assignments, the output of ChatGPT is more often correct than not. In our tests, ChatGPT can successfully generate code if we feed it:

• The entire assignment description, including context and background information;
• Just the example (terminal) output and ask it to reverse engineer that;
• Partial code and ask it to complete that.

The code provided by ChatGPT is often correct, but far from perfect. It misses documentation, comments, edge-cases, basic error handling or generates overly complicated code. Many also report that ChatGPT inherently fails to understand basic logic. But as mentioned earlier, the flawed code it generates is already able to pass common CS assessments.

Finally, another important limitation of ChatGPT (that is of lesser importance for generating code) is its “knowledge cutoff year”. ChatGPT is trained on data from before the year of 2021.