ChatGPT helps computer science instructors grading code.
May 3, 2023

5 ways CS instructors can benefit from ChatGPT in their workflow

In 30 seconds...

  • OpenAI's ChatGPT can be a great benefit to computer science teachers, we discuss 5 ideas to benefit from ChatGPT;
  • Many teachers use an autograder like CodeGrade, ChatGPT can be used to generate pattern matching (semgrep or RegEx) and unit tests for automatically grading student code;
  • Use ChatGPT to come up with example solutions for your assignment, so you can test and confirm your autograder and review potential misconceptions;
  • Let ChatGPT generate part of your content, we discuss what parts of your assignments can be easily generated by ChatGPT and which can't;
  • Finally: add a debug-ChatGPT assignment to your course to teach AI literacy;
  • This is the third article in our ChatGPT series, you can find our first article discussing the impact of AI to CS education here or read our second article discussing 5 assignment design ideas to create ChatGPT-proof assignments here.

ChatGPT has made a big impact on the field of education since its release in late 2022. ChatGPT and other AI-assistants help students easily generate code. Naturally, this has disrupted education and caused great challenges for academic integrity, especially in the field of Computer Science education. In our previous two articles on ChatGPT and its effects on CS education, we have first focused on the impact it has on education and second how coding course instructors can design ChatGPT-proof assignments.

Just looking at the problems ChatGPT causes for education seems just a bit too one-sided, as we believe current AI tools can also greatly help out in education. That’s why we are discussing 5 very practical ways that ChatGPT (or other AI-assistants like GitHub Copilot or Amazon’s CodeWhisperer) can actually benefit Computer Science instructors in their work. With rapidly increasing student numbers and an ever increasing teaching- and grading workload for coding teachers, we hope these tips and tricks can inspire you to add ChatGPT to your own workflow!

Want to learn more about ChatGPT in Coding Education? Join our live webinar on June 1st 2023, sign up below:

Benefit 1: Generating pattern matching

If your Computer Science course has a lot of practical assessments (which we believe most should have), you will probably spend a lot of time grading student code. Most teachers have adopted some form of autograding, because it helps reduce workload and because of its many pedagogical benefits (if done correctly). Autograding may consist of a simple local bash grading script, or an instant online autograding system like CodeGrade offers. But a big part of it is often pattern matching: either matching the student output for correctness or analyzing student code for specific patterns.

Two types of pattern matching are especially common in autograding:

  1. RegEx: teachers often use regular expressions (or RegEx) to parse students’ output. In this way, teachers can be more flexible in grading student output. They can check whether the output contains the right structure or the correct details, instead of the output having to be exactly right in its entirety (CodeGrade’s autograder already offers substring match with case insensitivity and ignore whitespace options as an easy way to to pattern match without a RegEx). 
  2. Semgrep: a great tool integrated with CodeGrade to help teachers analyze code structure statically. It is used by teachers to check if students use the right libraries, functions, programming methods and do not make common mistakes. You can watch our recent webinar or using Semgrep for education here.

Pattern matching gives CS instructors the ability to analyze code and output in a very flexible and powerful way. But coming up with a RegEx or Semgrep pattern can sometimes be challenging. This is where ChatGPT comes into play.

At its current stage, ChatGPT is very adequate in coming up with regular expressions. Simply describe your wishes for the RegEx or even give a template with placeholders like the example below and it will often come up with a correct RegEx (and even explain it!). It is always a good practice to first confirm the RegEx on a site like RegExr before publishing it to students in your autograder. With CodeGrade’s updated autograder, you can also instantly confirm your RegEx, read more on that here!

Where ChatGPT can already reliably create RegExes on its own, Semgrep patterns are still somewhat unpredictable. We did include this idea as we believe with the rapid developments and improvements, it’s only a matter of (short) time before ChatGPT can do that too. For now, consider ChatGPT as a companion for your Semgrep patterns and use the Semgrep Playground or CodeGrade’s support team to come up with your own.

Save time grading with CodeGrade and ChatGPT now!

Benefit 2: Generating Unit Tests

Unit Tests are another very common tool in the world of autograding. CodeGrade for instance supports many unit testing frameworks right out of the box, from JUnit to PyTest and from xUnit to Selenium. Teachers use these to check functionality of students’ code and with the right software, the output of a unit test can be made very student friendly.

If you have many assignments in your course, coming up with unit tests for all of these can be a lot of manual work. Luckily, ChatGPT (or other AI-assistants that can generate code) save the day! It is easy to prompt ChatGPT to write unit tests for the given code, in your preferred unit test framework. For an assignment in which students write a Passport class with some functions, we got the following output by prompting “Please write PyTest unit tests for this code: …”, followed by the entire example solution:

As we have already discussed in our previous article on ChatGPT, ChatGPT is a “bullshit-generator” (as coined by Princeton Computer Science professor Arvind Narayanan). However correct the generated unit tests may seem, there is never a guarantee that they are indeed correct and complete. You should always first thoroughly test and confirm them before publishing them in your code autograder.

Benefit 3: Generating Example Solutions

Most teachers will have a solution file for the assignments they create, this solution code or “answer key” is often used to test their autograder, generate example output and prepare for potential misconceptions. Especially testing an autograder is an important part of assignments nowadays, as you want your autograder to at least:

  • Be flexible enough to work with differences in student implementations;
  • Give a 100% score if the student code is completely correct;
  • Deduct partial points for smaller mistakes;
  • Be strict enough to catch missed edge-cases, error-handling and other minor mistakes;

This way, your autograder will encourage students and provide constructive feedback throughout their assignment.

To test your autograder on these points, only a correct example solution will not suffice. With a correct solution you can only confirm that your autograder will give a 100% score if correct, but you cannot check if it does point deductions as expected for incorrect or incomplete solutions.

This is where ChatGPT comes in handy! Instead of tasking your teaching assistants to come up with partial solutions, you can prompt ChatGPT to write code with mistakes (or simply task it to write code, as that often already does provide incomplete code as you can find here). See our ChatGPT prompt answer below:

As you can see, ChatGPT does a great job introducing mistakes to existing code and also explaining why these are mistakes. For this example, we simply provided the example solution source code and tasked it with: “Can you rewrite this code with intentional mistakes?”. Of course, you can play around with this: regenerating responses to get a larger variety of mistakes or explicitly mentioning the wanted mistakes in your prompt.

This method is a simple variant of “mutation testing” a common method of assessing the completeness of unit tests. We use “mutation testing” as one method to assess student written unit tests in CodeGrade, you may find more on assessing student unit tests in our documentation here!

Benefit 4: Generating content

This fourth benefit for teachers may be the most straightforward of them all, but is often overlooked still. ChatGPT is great at generating content for assignments. You can find countless articles online that discuss how you can use ChatGPT to generate written content, so we don’t want to repeat that. Instead, we do want to give some initial thoughts on generating course content with AI.

First of all, we do not believe AI can replace the teacher. A well thought-out curriculum with meaningful pedagogically correct assessments and content is still something that should be designed by a professional. However, within individual pieces of content, you can use ChatGPT to take some repetitive writing off your plate. For instance, to add optional background information or stories to “spice up” your assignments.

In the “Introduction to Python” course designed by CodeGrade, our assignments often consist of two parts: the actual code task and some interesting background information on the topic of the task. In an assignment on the Fibonacci sequence we for instance give some background on Leonardo Bonacci from Pisa or in the Parity Bit assignment we provide a short introduction on the importance of error detection in the world of telecommunication. This background information is by no means essential to finishing the assignment, but does provide context and is designed to spark a student’s interest.

As these pieces of background information are abundant in coding assignments but not essential, they can be very easily generated by ChatGPT. See the prompt below

What is important in these prompts is to tell ChatGPT to make it introductory, easy to read or fun. Our experience shows that background information from ChatGPT can otherwise very quickly become dry and technical, exactly the opposite of the goal of that text in your assignment!

Fun bonus idea: Who doesn’t remember that one teacher always slipping in jokes in every assignment. Everyone can become that teacher now with ChatGPT. Admittedly, the jokes are not always very funny, but taking a screenshot of ChatGPT trying to make a topical joke can already be funny in itself. See the (hilarious) example below:

Benefit 5: Add a debug-ChatGPT assignment to your course

As discussed in our previous article on 5 design ideas to create ChatGPT-proof coding assignments, we believe that it is important to consider incorporating ChatGPT in programming courses. In this way students become familiar with the strengths and weaknesses of AI assistants during coding and can create a healthy relationship with these tools continuing their (academic) careers.

One way to teach students about AI is a “debug-ChatGPT” assignment, as suggested by Orit Hazzan in her recent ACM opinion piece. We have discussed our thoughts on this assignment type in further detail in our previous article here.

ChatGPT brings a whole range of new assignment possibilities that teach AI literacy and let students practice with AI. Teaching students on AI literacy is a benefit for coding instructors and teaching assistants. When students are more literate and familiar with AI, their teachers can go from policing students on their use of AI to mentoring students on their use of AI.

Wrapping up…

Hopefully this article has inspired you to adopt ChatGPT in your own workflow, in a way that decreases your workload or simply takes away repetitive tasks from you. As mentioned before, we do not believe ChatGPT or other AI-assistants are able to take over many of the tasks of CS instructors. Human professionals are still very much needed to design meaningful curriculums, assessments and course content. However, we do believe ChatGPT can help with the individual creation of specific content in a course, from help with setting up and testing an autograder to generating content for individual assignments.

With all usages of AI, the publisher of the content is still responsible for the content. Please keep in mind that there are no guarantees that AI-generated code or text is correct and that you should always very thoroughly confirm them before publishing them. Especially in the field of education.

We also believe that developing a healthy relationship with AI is important for students and teachers alike. Becoming more literate with AI tools like ChatGPT means identifying when you can make use of AI in your own workflow. It also means being able to better identify when students may (ab)use AI in your CS course. Need help creating ChatGPT-proof code assignments? Read our previous article here!

We have seen that ChatGPT can greatly help you set up autograding quickly and effectively. CodeGrade offers the most powerful autograder for code education that is integrated directly in your LMS. Want to learn more about CodeGrade? Or would you like to chat with us to discuss the implications that ChatGPT has on your specific CS course? Shoot us an email at and we would love to call!

Devin Hillenius

Devin Hillenius


Continue reading

New in CodeGrade: Community Library

With the CodeGrade Community Library you can easily import fully automatically graded coding assignments into your own coding course.

Watch our ChatGPT and Coding Education webinar!

Watch CodeGrade's webinar on using ChatGPT in coding courses to help students work with this new tool and even use it in your grading worklfow yourself!

Join our webinar on ChatGPT in Coding Education!

Join CodeGrade’s CEO, Youri Voet, for a webinar on the impact of ChatGPT on computer science education. Learn how to make ChatGPT-proof coding assignments, teach AI literacy and how to use ChatGPT to set up automatic testing for your coding assignments.

New features: Assignment Schedules and Asynchronous Assignments

On May 8th and May 22nd, respectively, CodeGrade will launch two exciting new features. These features are Assignment Schedules and Asynchronous Assignments.

Sign up to our newsletter

Schedule a personalized tour of CodeGrade today