ChatGPT has been causing educators headaches since its release in late 2022. ChatGPT allows users to chat with an AI language model (”GPT-3.5” to be exact) in an easy-to-use environment for free. Naturally, this has disrupted education and caused great challenges for academic integrity. One of the highly affected fields is that of Computer Science, in which students can simply use ChatGPT or AI-assistants to generate code.
In our previous article, we discussed how ChatGPT can already help students generate code and solve coding problems (ChatGPT can easily pass the 2022 AP Computer Science A exam). Detecting AI-generated code is inherently hard, and as the language model keeps evolving, ChatGPT classifiers and plagiarism scanners will still remain very unreliable. Instead of banning ChatGPT, many teachers rather consider ChatGPT as the “next calculator” or “next Wikipedia”, in other words: design education that works with ChatGPT instead of against it.
In today’s article, we will discuss 5 code assignment design ideas that will make your computer science courses more ChatGPT-proof. These ideas “exploit” some of the known limitations of ChatGPT, which essentially is just a “bullshit generator” , a term coined by Princeton CS professor Arvind Narayanan. We also discuss how you can better prepare students for a future with AI. In an effort to help towards education that works with ChatGPT instead of against it.
Want to learn more about ChatGPT in Coding Education? Join our live webinar on June 1st 2023, sign up below:
Design idea 1: Inexplicit Edge Cases
From our internal research on using ChatGPT to generate code, we see that it does a very good job at solving (basic) programming assignments. As you can read in our previous article, ChatGPT has no problem coming up with the correct code if we feed it any of these:
- The entire assignment description, including context and background information;
- Just the example (terminal) output and ask it to reverse engineer that;
- Partial code and ask it to complete that.
What we did notice, however, is that the solution ChatGPT comes up with is often the most bare-bones solution. It fails to include edge cases and error handling, both often an important part of understanding a solution. See this example when tasked to create a Fibonacci generator:
The basics are correct: the code indeed does print the Fibonacci sequence. However, it misses essential edge-cases and input validation:
- Missed edge case: for n=0 and n=1 the output is incorrect, it will always print at least the first two elements of the fibonacci sequence.
- Missed input validation: it does not check if the user input is indeed an integer and it does not check if the input is indeed non-negative.
This is a relatively small code task, but the larger the task, the more potential edge cases ChatGPT will miss. Use this to your advantage: don’t explicitly mention required edge-cases in your descriptions, but instead only grade them using your autograder or unit tests. With an instant code autograder like CodeGrade, students will directly see the tests that passed and failed after submitting / running their code. This workflow also encourages students to continuously hand in intermediate submissions too. This will prevent students from using ChatGPT to come up with a full correct version instantly but instead discover these edge-cases on their own and come up with a solution themselves. These edge-cases simply show up as autograder tests and are not explicitly mentioned in the assignment description.
Using CodeGrade’s autograder? Let students interactively discover these edge-cases using our instant feedback (and online IDE). Want to challenge your students more? Simply make the edge cases more vague, so that your students know they missed “an edge-case” but have to discover which one this is themselves.
Design idea 2: Progressive Assignments
Traditionally, many computer science assignments look very similar: students get a document with one larger code task, sometimes including additional example output or code snippets. They then tackle this problem and hand in their code once before the deadline. This approach is very vulnerable to AI code generators: teachers have no way to inspect students progress and students lack feedback before the deadline. If they have only one assignment and one hand-in attempt, it is very easy to exploit other resources in coming up with that one solution.
With new interactive grading tools like CodeGrade, students can interact with an autograder (and optionally work in an online IDE). This has two large benefits against ChatGPT:
- Students can interact with the autograder and are encouraged to hand in subsequently before the deadline;
- Naturally, this allows teachers to get an overview of the progress made by the student.
We can use these benefits to improve our assignment design, going from a one-shot assignment to progressive assignments. In short:
- Give students a very brief and minimal assignment description to start with.
- Let students discover further assignment requirements and further sub-tasks using your autograder.
The benefit of this method over a traditional one-shot assignment is that students still need computational thinking and problem solving skills to solve an incremental assignment. With each additional sub-task, they have to first understand the additional requirements and then interpret how this should affect and change their code project. It is impossible to experience these problems if students simply get the entire code for a one-shot assignment generated through ChatGPT.
In the example of Fibonacci, we can have a very basic initial assignment description telling students to implement Fibonacci using the for-loop. After which, students are prompted with new subsequent sub-tasks using automatic tests and rubric categories in CodeGrade that only show up after they start handing in, see the AutoTest example from CodeGrade below:
Design idea 3: On-site exams with lockdown browser and IDE
We believe coding is a skill best learned by doing. Students need constant formative assessments throughout their academic career to acquire the coding skills they need. However, a great addition to these weekly or bi-weekly coding assessments can be an on-site final exam.
It is no surprise that it is easier to prevent students from using AI assisted coding tools in the controlled setting of an on-site final exam. What is hard, however, is to make sure students get the right coding environment during such an exam and that the logistics work for you as a course coordinator. This often required manually installing the right tools on hundreds of lab computers and afterwards manually collecting all submissions: a lot of administrative and logistic work for the teacher.
This does not mean we should go back to letting students code on paper (as former computer science students ourselves, we have been there), but instead make use of some novel online tools to reduce the administrative burden and let students code in a familiar environment. By combining CodeGrade’s IDE with a lockdown browser, you can control exactly which sites your students are and are not allowed to use. For instance: disabling ChatGPT but enabling useful resources like W3Schools or the official documentation of a used library. With CodeGrade’s IDE, students have a bare-bones but familiar coding experience, and with easy to set up instant autograding, they can validate and run (parts of) their code instantly during the exam.
.png)
