Autograde Python assignments step by step with CodeGrade.
February 8, 2022

Webinar: Automatically grading Python assignments in CodeGrade

In 30 seconds...

In this webinar we discussed:

  • Setting up a Python assignment in CodeGrade
  • How to use Input/Output tests to autograde Python
  • Autograding Code Quality of Python assignments
  • How to use the Unit Test step
  • More advanced Python autograding in CodeGrade
  • How to use Friendly for better error messages

Learn all about it in this article or watch the webinar here!

In our latest webinar, we tell you everything you need to know to get started with Python assignments in CodeGrade. See how you can automatically grade Python assignments in CodeGrade using simple I/O tests or more powerful tools such as the Unit Test or Code Quality test steps. Additionally, we dive deeper into some advanced testing methods such as Semgrep and ways to customize your Unit tests. Finally, find out about friendly-traceback, a great external python tool for making python error tracebacks more friendly for beginners! This webinar was part of our monthly CodeGrade Webinars series and was recorded live on February 4th 2022 - available on-demand now.

Why CodeGrade is the best place to autograde Python assignments

CodeGrade comes with Python2 and 3 by default, no installation necessary! That means you can run your students’ Python code as you would in your terminal. Combining this with the I/O test step makes autograding python assignments a breeze. Not enough? CodeGrade’s Unit Test step comes pre-installed with PyTest and our Code Quality step includes flake8 and pylint. AutoTest’s plug and play interface mean that while you can use most of these pre-installed frameworks without any configuration, you can also install any custom tools, modules, or configurations that you like!

Before we can use all these exciting features we first need to set up our assignment settings by making sure our students know what they need to hand in and what they are being tested on.

Setting up a Python assignment in CodeGrade

Once we’ve created our assignment, whether that’s in an LMS like Canvas or Blackboard, or in a standalone version of CodeGrade, we need to configure our assignment’s general settings. Firstly, we need to make sure that our deadline has been set and our Assignment state is set to “Hidden” while we are creating our assignment (Keep in mind the deadline might have to be set in your LMS). Next, we can check our submission settings and, if necessary, adjust the number of times a student can submit a solution, or allow them to submit their solutions through their own github repositories.

With our submission settings done we can move on to the Hand-In requirements. This setting allows you to specify the files your student is required to hand in and prevent them from submitting anything else. We can do this by, first, denying all files by default and then specifying the required filename. For instance, ``.

Finally, we need to set up our Rubrics. There are two types of rubric categories, Continuous and Discrete. We typically recommend using discrete rubric categories because you can specify in more detail what your students need to do in order to get each point but in some cases, continuous rubrics can be more flexible, especially when automatically grading assignments. Don’t forget that each description field accepts Markdown so you can make them as clear as possible for students.

How you can use CodeGrade’s I/O test step to autograde you Python assignment

The I/O test step is an integrated feature in CodeGrade’s AutoTest. It allows you to interact with your student’s program as if you were working directly in your terminal and is a great way to test simple Python scripts. We can use the I/O test in different ways depending on whether your student’s code is static, meaning it has a hard coded output, or dynamic, meaning it needs to be interacted with to produce an output.

For static code, we can simply leave the Input field empty and check whether the script returns the expected output. For dynamic programs, we can interact with the code using the input field to test different sets of input and see how our students’ code handles them.

When designing our tests we should consider whether we want to test the entire script or just a specific part of it. In the case of the former, we can simply run the program in the “Program to test” field by entering `python3`. For the latter,, if we just want to test a function or a stored variable we can call these by importing our students’ code using the command `python3 -ic “import your_script”`. The `-i` flag will allow us to use the input field as if we were using the Python interpreter in our own console. We can then access, for example, a function by entering  `your_script.your_function(input1, input2)` in the input field.

Basic autograding set up for a Python assignment

Autograding Code Quality of Python assignments in CodeGrade

CodeGrade’s AutoTest has a dedicated Code Quality check step. Using this test step we can check the style errors, inefficient or unused code, and other common coding mistakes to name a few. Learning to work with a style guide is an essential skill for CS students and one of the most common style guides for Python is the PEP8 Style Guide. We can enforce the use of the PEP8 style guide with the flake8 static code analysis tool which is installed in the Code Quality test step by default. Unlike other linters, flake8 comes with PEP8 pre-configured meaning there are no extra configuration steps necessary. However, you can of course ignore specific rules by entering the command `--ignore=<your_rule>` in the Extra arguments field.

Alternatively, we could use pylint which also comes pre-installed in the Code Quality test step. It does require some configuration but it provides more flexibility and enables you to create your own custom style guide.

Importing flake8 for Code Quality

Ease the burden of grading with CodeGrade's sophisticated and flexible autograder!

Using CodeGrade’s Unit Test step for Python assignments

PyTest is one of the most popular unit testing frameworks for Python. CodeGrade has its own PyTest wrapper script that is readily available in the drop-down menu of the Unit Test step. CodeGrade’s PyTest wrapper parses the output xml from your tests and displays each test suite as its own individual test step in the AutoTest. To use pytest we first have to create our own unit test script using the PyTest module and upload it as a fixture. With our script available in our test environment we can then run it in a Unit Test step by entering the file path `$FIXTURES/` under Files to test.

Using PyTest for Unit Testing

Advanced Python autograding: Semgrep, custom PyTest names and weights

As you can see in the example above, each test suite has its own name, which is the name we use when we define our test in the test script. However, by passing `record_xml_attribute` as an argument to our test definition, we can specify different names and also set a different weight. These are parsed by the PyTest wrapper and are then set for each test in CodeGrade.

Advanced Python Autograding

Sometimes we want to check that students are using a specific built-in function or that they’re looping over a certain array. Thanks to Semgrep, we’re able to check our students’ code for specific code structures by using pattern-matching. Semgrep is installed in CodeGrade’s Unit test step so we can use it by simply uploading our code-structure rules as a YAML file and selecting it in the Unit test step.

Friendly for better error messages

Finally, Friendly is an external Python module that extends Python’s default error traceback, making it more readable for beginner Python developers. Friendly is easy to install in CodeGrade by simply uploading the script below as a fixture and installing it in the Global Setup script in the setup portion of AutoTest. With Friendly installed, students get better tracebacks when their code is run and gives them a deeper understanding of what went wrong, so they can then fix their code.

Error traceback with Friendly

Samuel Natarajan

Samuel Natarajan

Teacher Success Manager
Meet Sam, a music enthusiast! He meticulously ensures CodeGrade clients excel and deeply understand the product, creating an unforgettable experience. He sees CodeGrade's cultural diversity and vibrancy as more than just a workplace.

Continue reading

How to Stop Caring about Grades

Learn to save time with grading with alternative methods for teaching. Make your course accessible and inclusive with methods backed by research.

How to Stop Caring about Grades

Embracing a New Era: Harnessing Generative AI for Computer Science Education

Discover the transformative impact of Generative AI (Gen AI) on computer science education. Explore innovative tools and methodologies reshaping learning experiences and preparing learners for an AI-centric future. Uncover benefits, challenges, and evolving practices at the intersection of technology and pedagogy.

Coding Quizzes are here!

Discover the latest addition to CodeGrade: coding quizzes! Elevate your learning experience with dynamic assessments seamlessly integrated with AutoTest V2. Engaging, clear, and user-friendly. Learn more now!

Coding Quizzes are here!

Transforming CS Education: Innovative Strategies for Curriculum Enhancement

Discover how Experiential Learning Theory transforms traditional teaching methods and improves computer science curriculum for optimal student engagement and success.

Sign up to our newsletter

See how CodeGrade can transform your courses today!