Autograding C# with xUnit
Guides
December 28, 2020

Setting up automated grading for C# with unit testing in CodeGrade

Sometimes a request from a customer inspires me to set up a CodeGrade assignment and explore new ways of autograding. In this case I got a request to provide an example of how to do unit testing to autograde a C# assignment. Creating C# assignments in CodeGrade is rather easy, either using mono, or by actually using .NET Core, which is available for Linux and MacOS as well nowadays. The thing is, though, often you don't want students to hand in a complete project, as that makes it harder to focus on the important stuff: the actual files they create. It is much easier and less error prone, especially for basic assignments, if students just hand in the .cs files they have created instead of the whole Visual Studio project.

Now comes the interesting part: using those files, and automatically creating the project structure around them. The reason we want to do this, is so that we can use the .NET tools to test the code and write unit tests using xUnit, one of the most popular unit testing frameworks for C#. The .NET Core framework provides the "dotnet" command to run projects and tests. But, for the "dotnet" command to work it has to be a valid project. In this case, we do not want students to hand in full projects, so we will have to craft the project and insert the tests automatically. This might sound difficult, but it is actually not that hard and quite fast to all set up.

Before we start setting up the CodeGrade assignment straight away, it is usually a good idea to start locally on your own computer. There we can easily figure out how we can create the structure, test if it works, and then port it to CodeGrade when we have a clear and clean idea.

For this particular example, we will be following most of the structure from this excellent guide from Microsoft. In this example, students will create a small program that will check if a number is prime. Following the earlier guideline, students will only hand in the PrimeService.cs file, which is a simple function to determine if a number is a prime. To make sure students only hand in this file, we can use CodeGrade's hand-in instructions. The Microsoft guide walks you through setting up a whole solution, but for our purposes, that's not necessary. The simple structure we would like to create is as follows:

PrimeService/

  • PrimeService.cs: this is the file that students hand in.
  • PrimeService.csproj: this is a fixture we generate and then upload to AutoTest.

PrimeService.Tests/

  • PrimeService_IsPrimeShould.cs: our xUnit tests live here.
  • PrimeService.Tests.csproj: this is a fixture we upload.

This gives us the following tasks:

  1. Generate the .csproj files
  2. Write the unit tests
  3. Create a simple script that generates the folder structure.
  4. Port it to CodeGrade.

Generating the .csproj files

Let's start with the first step, and that's creating the .csproj files. We can just execute some simple dotnet commands to create these:

dotnet new classlib -o PrimeService

This command will create the PrimeService.csproj file (and some other stuff including the directory structure, but we can disregard that).

Write the unit tests

Now let's quickly take a look at the solution file that students will hand in:

using System;

namespace Prime.Services
{
    public class PrimeService
    {
        public bool IsPrime(int candidate)
        {
            if (candidate < 2)
            {
                return false;
            }

            for (var divisor = 2; divisor <= Math.Sqrt(candidate); divisor++)
            {
                if (candidate % divisor == 0)
                {
                    return false;
                }
            }
            return true;
        }
    }
}

Then we can start writing some unit tests for this. First, we'll have to create the tests project and link them to the solution:

dotnet new xunit -o PrimeService.Tests
dotnet add ./PrimeService.Tests/PrimeService.Tests.csproj reference ./PrimeService/PrimeService.csproj

Then within this directory, we can create the actual unit test. For this example let's just create one simple test:

using Xunit;
using Prime.Services;

namespace Prime.UnitTests.Services
{
    public class PrimeService_IsPrimeShould
    {
        [Fact]
        public void IsPrime_InputIs1_ReturnFalse()
        {
            var primeService = new PrimeService();
            bool result = primeService.IsPrime(1);

            Assert.False(result, "1 should not be prime");
        }
    }
}

We can already test this on our solution locally by simply running the "dotnet test" command.

Want to start using CodeGrade in your course?

Generating the folder structure

Once we are done with writing tests, we can start porting this to CodeGrade, but first we will have to write a simple script that creates the desired directory structure from the student files. This will look like this:

#!/usr/env/bin bash

mkdir PrimeService PrimeService.Tests
mv PrimeService.cs PrimeService
mv $FIXTURES/PrimeService.csproj PrimeService
mv $FIXTURES/PrimeService_IsPrimeShould.cs PrimeService.Tests
mv $FIXTURES/PrimeService.Tests.csproj PrimeService.Tests

# Install the XunitXML logger, so it works with the CodeGrade unit test step type.
cd PrimeService.Tests
cg-xunit install
dotnet add package XunitXml.TestLogger
dotnet restore

In the first lines, we are created the directories and are moving the fixtures (which we will soon upload in the autograder configuration) to the student directory. This will create our folder structure.

Those last 4 lines might look a little bit confusing and we found this out after a bit of debugging. As we want to use CodeGrade's "Unit Test" step type, we need to install the XunitXML logger and add it to the project, we are also doing an cg-xunit install (the wrapper script to easily use Xunit with CodeGrade) to install all dependencies.

Creating the AutoTest in CodeGrade

Now we are ready to create our CodeGrade assignment. Simply upload the generated project files, the tests and the student setup script as your AutoTest fixtures and then execute the student_setup.sh in the "Per-student setup script". Finally create a category with your unit test and execute the following command in this:

cd PrimeService.Tests; cg-xunit run

Now the tests will automatically run as soon as a student hands in. Make sure to upload a test submission, so that you can check if the AutoTest works as planned.

The most asked question I get is how long it takes to set up the autograding. So let me answer this question for you for this example. It took me just under an hour to figure out what kind of directory structure was necessary, writing the setup script and testing and debugging it locally and on AutoTest. The upside is that this is work I only have to do once, and for all other assignments I can use the same scripts and easily import this part of the AutoTest. The only thing that has to change are the actual unit tests that I will have to write.

If you have any questions about this example in AutoTest or about CodeGrade in general, feel free to email me at youri@codegrade.com and I am more than happy to help you.

Continue reading

Start using CodeGrade now to supercharge your feedback on code.