Autograding Jupyter Notebooks tutorial webinar for computer science education
Guides
December 6, 2021

Webinar: Grading Jupyter Notebooks

In 30 seconds...

In this webinar we discussed:

  • What is a Jupyter Notebook?
  • Grading Jupyter Notebooks manually
  • Automatically running a Jupyter Notebook
  • How to format your Jupyter Notebook template for autograding?
  • Autograding Jupyter Notebooks with I/O Tests
  • Advanced Jupyter Notebook grading

Learn all about it in this article or watch the webinar here!

In our latest webinar, we tell you everything you need to know to get started with Jupyter Notebooks (also known as IPython Notebooks) in CodeGrade. We explain how Jupyter Notebooks are structured and how you can best manually grade and give feedback to them using CodeGrade's inline feedback, general feedback and render viewer. Furthermore, we teach you how you can effectively and easily autograde the Python code in Jupyter Notebooks. This webinar was part of our monthly CodeGrade Webinars series and was recorded live on December 3rd 2021, it is available on demand now.

Jupyter Notebook theory

Jupyter Notebooks, formerly known as IPython Notebooks, are files with the `.ipynb` extension. Even though Jupyter Notebooks support multiple programming languages, we most often see them used for Python courses. They combine these "code cells" with text cells (supporting Markdown and LaTeX). This combination makes for a very powerful educational tool that is used not only for courses like Introduction to Python, but also more advanced courses like Machine Learning, Data Science and Computer Vision.

On the inside, a Jupyter Notebook is simply a JSON document. All cells, metadata and output are stored in JSON. It is very important to understand that the output is also statically saved here and can thus be altered manually or be outdated if a Jupyter Notebook has not been run.

Manually Grading Jupyter Notebook

CodeGrade supports Jupyter Notebooks right out of the box and renders the notebook for you. You can then use all of CodeGrade's feedback features to give feedback and grade the Jupyter Notebook manually. You can give feedback on individual cells, lines of code and general feedback over the Jupyter Notebook as a whole.

Giving feedback to a text cell in CodeGrade: hover over the text cell and click the feedback button.

Automatically running Jupyter Notebooks for manual grading

As mentioned before: the output cells are also saved in the Jupyter Notebook JSON document. This means that when we are grading a Jupyter Notebook handed in by the student in CodeGrade, we are not guaranteed to have the latest version of the output (it can be manually altered by the student or the notebook could not be run). To solve this, we can use CodeGrade AutoTest to automatically run each IPython Notebook immediately after they are handed in. An up-to-date version of the Jupyter Notebook will then show up in the Code Viewer under the AutoTest output:

An automatically run Jupyter Notebook in CodeGrade's Code Viewer.

We can do this with a very simple command in a Run Program step in AutoTest. Add the following command (explained in more detail in the webinar):

-!- CODE language-sh -!- jupyter nbconvert --execute --to notebook 
-–output $AT_OUTPUT/jupyter.ipynb $STUDENT/jupyter.ipynb -–allow-errors

Start autograding your Jupyter Notebook assignments easily and effectively with CodeGrade!

Formatting a Jupyter Notebook for easy and effective autograding

The easiest and most common way to autograde Jupyter Notebooks is to convert them to code, i.e. appending all code cells to form a regular Python script. In CodeGrade, we can then assess this Python script like we would normally do.

There are a couple of ways to easily assess multiple assets of a Jupyter Notebook assignment in CodeGrade:

  • We can assess the content of variables for correctness
  • We can assess the correct working of functions
  • We can run the whole sequence of code cells and check the output as a whole

In the webinar, we show you step by step how you can set up an AutoTest in CodeGrade to autograde these parts of a Jupyter Notebook assignment. What is important to note is that you need to know the names of the variables and functions to be able to assess them. A good practice is to provide your students with a template Jupyter Notebook file with predefined variables and functions that they can fill in themselves.

Autograding Jupyter Notebooks in CodeGrade

Autograding Jupyter Notebooks in CodeGrade becomes easy when formatting your assignment as explained above. The steps to do so are shown in the webinar, to summarize:

  1. Convert the `.ipynb` file to a Python script in a Run Program step using `jupyter nbconvert --to script jupyter.ipynb`. (change the name `jupyter.ipynb` to the name of your Jupyter Notebook assignment, you can enforce this using Hand In Requirements)
  2. Create an I/O Test in the same that runs `python -i jupyter` (change the name `jupyter` to the name of your assignment, this is the file you generated in the step above).
  3. In this I/O test, you can now use regular Python as input and assess individual parts of the assignment. For instance, printing a variable or running a function and printing its result. You can write the expected output in the "Expected Output" field.
A test category for autograding Jupyter Notebooks

Ignoring superfluous print statements

One thing to be aware of is that we are checking the stdout of the scripts, which are run completely when importing. As a result, students can clutter the output with additional print statements outside of functions. There’s two ways to prevent this:

  1. Importing the script with the stdout redirected. This can be done using this little code snippet, which you run via `python3 -i import_without_print.py`:

-!- CODE language-python -!-from contextlib import redirect_stdout
from os import devnull

with redirect_stdout(open(devnull, 'w')):
    import jupyter #<-- The name of your script


  1. If you are providing Jupyter Notebook skeleton code to your students, you can make sure to add a `if __name__ == "__main__":` statement every time you print solutions. This way, the students can see their solutions while interacting with the notebook, but these solutions will not be printed when importing the code (using the script above or with a regular import).

The above just lists a brief overview of what is discussed in the webinar. In the webinar, we went into detail on all things grading Jupyter Notebooks and went over all the steps on how to reproduce this for your assignment. We are here to make your course with CodeGrade a success, so please do not hesitate to reach out to us on Discord or via support@codegrade.com with any question, feature request or if you would like some technical or educational consultancy. We are always more than happy to help you out!

Devin Hillenius

Devin Hillenius

Co-founder, Product Expert
Devin is co-founder and Product Expert at CodeGrade. During his studies Computer Science and work as a TA at the University of Amsterdam, he developed CodeGrade together with his co-founders to make their life easier. Devin supports instructors with their programming courses, focusing on both their pedagogical needs and innovative technical possibilities. He also hosts CodeGrade's monthly webinar.

Continue reading

Teaching Intro to Python with CodeGrade

The CodeGrade Introduction to Python course is an 8-week basic Python course. Students are not required to have any prior knowledge on programming or Python. This course will cover the basic concepts of programming up to Python specific modules and OOP design. It is available for all instructors now.

New release CodeGrade QuietStorm.1!

Find out about all the features and updates coming with our latest release, QuietStorm.1!

New release CodeGrade QuietStorm.1!

Automatically grading Haskell code assignments

Learn about autograding Haskell coding assignments for Computer Science education courses. CodeGrade can help you use tools like input and output checking, Quickcheck, Tasty and HUnit unit test autograding, HLint code quality checking and code structure autograding using semgrep.

Top tips for teaching programming

The most efficient ways to teach students how to code, by defining your target audience, designing your assignments in a meaningful way and eliminating distractions in the classroom.

Top tips for teaching programming

Learn more about CodeGrade!

Grow your coding classroom
without compromise.