How to use custom test blocks

The Custom Test block is one of the most flexible tools in CodeGrade’s AutoTest system. It lets you write your own grading script in any language (like Bash or Python), so you’re not limited to predefined test types like input/output checks or basic unit tests.

Your script can perform any evaluation logic you need, and then output a JSON object to communicate the results back to CodeGrade. For example:

{ "tag": "points", "points": "3/5" } 

That line means the student earned 3 out of 5 points—and that’s all CodeGrade needs to assign a grade.

This combination of scripting freedom and simple output formatting makes Custom Tests a great fit for assignments that require a little extra complexity.

Why Use a Custom Test Block

There are several situations where the standard AutoTest blocks just don’t cut it. Here’s when a Custom Test block really shines:

1. Complex or Multi-Step Evaluations

Maybe the assignment requires generating a file, checking its contents, validating a format, and then scoring based on that. A Custom Test lets you script all those steps in one place.

This is perfect for tasks that go beyond a single input/output comparison or can’t be fully captured by a unit test framework.

2. Custom Grading Logic

Want to award partial credit? Or maybe you have your own scoring rules based on specific criteria?

With a Custom Test, you can write the exact logic you need. For example, you can check whether each part of the output meets expectations and assign points incrementally, rather than using all-or-nothing scoring.

3. External or Specialized Tools

If your assignment involves custom tools, external libraries, or even a specific linter or data analysis framework, no problem.

You can run those tools inside your script. Since the Custom Test doesn’t rely on built-in test runners, it gives you full control.

4. When Other Blocks Don’t Fit

Sometimes, an assignment just doesn’t fit any of the built-in templates. That’s where Custom Test becomes your go-to.

In short: if you can script it, you can grade it.