Automatically grading Python programming tasks for CSEd classrooms
March 17, 2022

Automatically grading Python code assignments

In 30 seconds...

  • Python is a popular programming language in education, especially for introductory courses and applied scripting or data science courses. 
  • With the right tools, it is very easy and beneficial to autograde Python.
  • Learn about Python I/O Testing, PyTest Unit Testing for education, Code Quality Grading with Flake8 and PyLint and Python Code Structure grading.

The need for Python autograding

Coding is slowly but surely becoming part of every professional field, making a fundamental knowledge of programming extremely useful. And with computer science classes being introduced in K-12/primary education, we’re sure to see an even bigger increase in Computer Science students, especially in Python courses.

With that being said, teachers can also expect a higher workload. Unzipping, downloading, running, adding feedback and re-uploading submissions per student and per assignment is a great burden! Unfortunately, this can take time away from teaching and giving students personal attention. This is where autograding can really help! 

Not only does automatic grading help teachers, but it also directly benefits students. Firstly, with an instant feedback mechanism, students can submit their work and see immediately where to improve. They can correct their code and resubmit, and this feedback loop continues. This quick turnaround helps students learn at a consistent pace - no longer do they need to wait a few days for their answers, only to forget how they did it in the first place. Moreover, with CodeGrade, students can also directly code in our integrated editor (directly from within the LMS) or hand work in directly through their GitHub or GitLab repository and get their feedback instantly! 

Secondly, after handing in, students can see if their solution works and fix it if it doesn’t, making sure you will never have to fix their compiler errors again. Finally, you have more time to focus on the interesting and important facets of teaching. For example, cutting down hours of grading time allows educators to give more attention to student questions and wider learning objectives.

Ideally, an autograder balances two elements; ease of use - students and teachers should be able to get to grips with the autograder quickly and easily; and flexibility - as learners move from basic programming assignments to more advanced ones, an autograder should function with a wider range of accepted outputs, among other nuances. 

Setting up a basic autograded Python, Jupyter Notebook or IPython Notebook assignment is pretty simple. We’ve outlined some of the options below.

Input/Output autograding

Many programming courses, particularly introductory courses, teach students how to create programs and interact with them through the command line interface (CLI). For instance, a simple program might prompt a user to provide a number and, in return, print a statement telling the user that their number is odd or even. Input/Output (I/O) tests are a great way of checking that these simple programs work as expected. However, this would traditionally require teachers to download each of their students’ submissions and run them independently in their local environment, putting inputs to the program one at a time and making themselves vulnerable to human error.

Fortunately, CodeGrade’s I/O test step allows you to run a program as you would on your own CLI - enter an input (either standard input, stdin, or as input argument) to the program, and test that the predicted output matches the actual output. With CodeGrade, this doesn’t have to be an exact match, but can be made more flexible by the teacher by modifying case sensitivity, substring matching, whitespace sensitivity or by using a regex.

CodeGrade is the most flexible and easy to use autograder for Python code assignments. Use it in your classroom!

Unit Test autograding

Once courses begin to venture into more complex programming assignments, particularly those with functions and other encapsulated code, unit testing is a natural tool for autograding. Unit testing frameworks often enable multiple ways of testing that aren’t just limited to matching a predicted outcome with an actual outcome. For Python, CodeGrade has support for PyTest out of the box. But frameworks like PyUnit can be installed too.

CodeGrade’s dedicated Unit Test Step parses the XML output of your unit tests and displays each test individually and checks whether or not they passed. The Unit test step makes customizing your unit tests really easy, enabling you to write custom descriptions for each test, weighting them differently, and even providing hints when the tests fail!

Code Quality autograding

While coding classrooms mostly cover the syntax of programming languages and the functionality of functions, procedures and modules, one thing that is sometimes overlooked is the readability and quality of code. Instantly autograding Code Quality or Style directly shows students where they can improve their code inside their learning environment. This can help students use better variable and function names, use their whitespace more consistently and prevent other common coding bad practices. 

The Code Quality test step in CodeGrade’s AutoTest makes a host of industry-standard linters and static code quality assessment tools available for teachers to automatically grade code quality and formatting! For Python, CodeGrade offers linters such as Flake8 and PyLint. While some configurations are available by default, CodeGrade offers you the ability to upload your own custom style guides and rules to give you full control. Custom style guides allow you to select which rules to check and alter the severity level of each infraction. The Code Quality step doesn’t just provide a grade, it also automatically highlights your students' code directly on the lines where the mistakes were identified, making it simple and intuitive for students to understand what went wrong and how to fix it.

Code Structure autograding

A type of autograding that is exclusively offered by CodeGrade’s AutoTest is easy code structure checking. For this, we have implemented our own version of the powerful Semgrep tool. 

With this, you can now not only check if your students’ code works and whether the style is correct, but you can also check if they actually solved your coding challenge in the way that you want. You can automatically check if they use prohibited libraries, if they actually use a for-loop and not a while-loop and if they did not hardcode their answer but rather solved the problem. What’s more, you can also make sure that students define the correct variables, and functions, to make sure they get a helpful warning if they did not.

Devin Hillenius

Devin Hillenius

Co-founder, Product Expert
Devin is co-founder and Product Expert at CodeGrade. During his studies Computer Science and work as a TA at the University of Amsterdam, he developed CodeGrade together with his co-founders to make their life easier. Devin supports instructors with their programming courses, focusing on both their pedagogical needs and innovative technical possibilities. He also hosts CodeGrade's monthly webinar.

Continue reading

New in CodeGrade: Community Library

With the CodeGrade Community Library you can easily import fully automatically graded coding assignments into your own coding course.

Watch our ChatGPT and Coding Education webinar!

Watch CodeGrade's webinar on using ChatGPT in coding courses to help students work with this new tool and even use it in your grading worklfow yourself!

Join our webinar on ChatGPT in Coding Education!

Join CodeGrade’s CEO, Youri Voet, for a webinar on the impact of ChatGPT on computer science education. Learn how to make ChatGPT-proof coding assignments, teach AI literacy and how to use ChatGPT to set up automatic testing for your coding assignments.

New features: Assignment Schedules and Asynchronous Assignments

On May 8th and May 22nd, respectively, CodeGrade will launch two exciting new features. These features are Assignment Schedules and Asynchronous Assignments.

Sign up to our newsletter

Book a quick 30-minute demo and see how it works for you!