5 ways CS instructors can benefit from ChatGPT in their workflow

Benefit 2: Generating Unit Tests

Unit Tests are another very common tool in the world of autograding. CodeGrade for instance supports many unit testing frameworks right out of the box, from JUnit to PyTest and from xUnit to Selenium. Teachers use these to check functionality of students’ code and with the right software, the output of a unit test can be made very student friendly.

If you have many assignments in your course, coming up with unit tests for all of these can be a lot of manual work. Luckily, ChatGPT (or other AI-assistants that can generate code) save the day! It is easy to prompt ChatGPT to write unit tests for the given code, in your preferred unit test framework. For an assignment in which students write a Passport class with some functions, we got the following output by prompting “Please write PyTest unit tests for this code: …”, followed by the entire passport.py example solution:

As we have already discussed in our previous article on ChatGPT, ChatGPT is a “bullshit-generator” (as coined by Princeton Computer Science professor Arvind Narayanan). However correct the generated unit tests may seem, there is never a guarantee that they are indeed correct and complete. You should always first thoroughly test and confirm them before publishing them in your code autograder.

Benefit 3: Generating Example Solutions

Most teachers will have a solution file for the assignments they create, this solution code or “answer key” is often used to test their autograder, generate example output and prepare for potential misconceptions. Especially testing an autograder is an important part of assignments nowadays, as you want your autograder to at least:

• Be flexible enough to work with differences in student implementations;
• Give a 100% score if the student code is completely correct;
• Deduct partial points for smaller mistakes;
• Be strict enough to catch missed edge-cases, error-handling and other minor mistakes;

This way, your autograder will encourage students and provide constructive feedback throughout their assignment.

To test your autograder on these points, only a correct example solution will not suffice. With a correct solution you can only confirm that your autograder will give a 100% score if correct, but you cannot check if it does point deductions as expected for incorrect or incomplete solutions.

This is where ChatGPT comes in handy! Instead of tasking your teaching assistants to come up with partial solutions, you can prompt ChatGPT to write code with mistakes (or simply task it to write code, as that often already does provide incomplete code as you can find here). See our ChatGPT prompt answer below:

As you can see, ChatGPT does a great job introducing mistakes to existing code and also explaining why these are mistakes. For this example, we simply provided the example solution source code and tasked it with: “Can you rewrite this code with intentional mistakes?”. Of course, you can play around with this: regenerating responses to get a larger variety of mistakes or explicitly mentioning the wanted mistakes in your prompt.

This method is a simple variant of “mutation testing” a common method of assessing the completeness of unit tests. We use “mutation testing” as one method to assess student written unit tests in CodeGrade, you may find more on assessing student unit tests in our documentation here!

Benefit 4: Generating content

This fourth benefit for teachers may be the most straightforward of them all, but is often overlooked still. ChatGPT is great at generating content for assignments. You can find countless articles online that discuss how you can use ChatGPT to generate written content, so we don’t want to repeat that. Instead, we do want to give some initial thoughts on generating course content with AI.

First of all, we do not believe AI can replace the teacher. A well thought-out curriculum with meaningful pedagogically correct assessments and content is still something that should be designed by a professional. However, within individual pieces of content, you can use ChatGPT to take some repetitive writing off your plate. For instance, to add optional background information or stories to “spice up” your assignments.

In the “Introduction to Python” course designed by CodeGrade, our assignments often consist of two parts: the actual code task and some interesting background information on the topic of the task. In an assignment on the Fibonacci sequence we for instance give some background on Leonardo Bonacci from Pisa or in the Parity Bit assignment we provide a short introduction on the importance of error detection in the world of telecommunication. This background information is by no means essential to finishing the assignment, but does provide context and is designed to spark a student’s interest.

As these pieces of background information are abundant in coding assignments but not essential, they can be very easily generated by ChatGPT. See the prompt below

What is important in these prompts is to tell ChatGPT to make it introductory, easy to read or fun. Our experience shows that background information from ChatGPT can otherwise very quickly become dry and technical, exactly the opposite of the goal of that text in your assignment!

Fun bonus idea: Who doesn’t remember that one teacher always slipping in jokes in every assignment. Everyone can become that teacher now with ChatGPT. Admittedly, the jokes are not always very funny, but taking a screenshot of ChatGPT trying to make a topical joke can already be funny in itself. See the (hilarious) example below:

Benefit 5: Add a debug-ChatGPT assignment to your course

As discussed in our previous article on 5 design ideas to create ChatGPT-proof coding assignments, we believe that it is important to consider incorporating ChatGPT in programming courses. In this way students become familiar with the strengths and weaknesses of AI assistants during coding and can create a healthy relationship with these tools continuing their (academic) careers.

One way to teach students about AI is a “debug-ChatGPT” assignment, as suggested by Orit Hazzan in her recent ACM opinion piece. We have discussed our thoughts on this assignment type in further detail in our previous article here.

ChatGPT brings a whole range of new assignment possibilities that teach AI literacy and let students practice with AI. Teaching students on AI literacy is a benefit for coding instructors and teaching assistants. When students are more literate and familiar with AI, their teachers can go from policing students on their use of AI to mentoring students on their use of AI.

Wrapping up…

Hopefully this article has inspired you to adopt ChatGPT in your own workflow, in a way that decreases your workload or simply takes away repetitive tasks from you. As mentioned before, we do not believe ChatGPT or other AI-assistants are able to take over many of the tasks of CS instructors. Human professionals are still very much needed to design meaningful curriculums, assessments and course content. However, we do believe ChatGPT can help with the individual creation of specific content in a course, from help with setting up and testing an autograder to generating content for individual assignments.

With all usages of AI, the publisher of the content is still responsible for the content. Please keep in mind that there are no guarantees that AI-generated code or text is correct and that you should always very thoroughly confirm them before publishing them. Especially in the field of education.

We also believe that developing a healthy relationship with AI is important for students and teachers alike. Becoming more literate with AI tools like ChatGPT means identifying when you can make use of AI in your own workflow. It also means being able to better identify when students may (ab)use AI in your CS course. Need help creating ChatGPT-proof code assignments? Read our previous article here!

We have seen that ChatGPT can greatly help you set up autograding quickly and effectively. CodeGrade offers the most powerful autograder for code education that is integrated directly in your LMS. Want to learn more about CodeGrade? Or would you like to chat with us to discuss the implications that ChatGPT has on your specific CS course? Shoot us an email at hello@codegrade.com and we would love to call!