Use Selenium for webdev courses to autograde website UI and UX in CodeGrade
Guides
July 21, 2021

Autograding web user interfaces with Selenium and Jest in CodeGrade

Web applications are a booming business nowadays. It is no surprise that web development courses are a core part of most current computer science programs, in addition to being present in most multimedia and design programs too. CodeGrade itself is a web application that was initially developed as part of a software engineering course at the University of Amsterdam.

Autograding websites is a little less straightforward, however. Often, the main aspect to grade is the user experience or user interface that the students have created using languages like HTML, CSS and JavaScript or using one of the many web frameworks available. For this, we cannot rely on (command line) input and output tests or unit testing like we usually do. Instead, we will have to look into ways to automatically interact with their website UI and test that.

This is where the two powerful tools come into play: Jest and Selenium. Jest is one of the most popular (unit) testing frameworks for JavaScript (and installed by default on CodeGrade) and Selenium is an open source tool that automates browser testing.

In this blog, we will discuss how you can set up a basic Jest and Selenium configuration in CodeGrade to autograde web development assignments. We will use some information gathered by Johan Holmberg from Malmö University, who has shared his steps to set up Selenium in CodeGrade in a public GitHub repository.

Installing Jest and Selenium in CodeGrade

To start autograding the UI of your web development assignments, we will need two main tools: Jest and Selenium.

  • Jest is a testing framework for JavaScript. It currently is one of the most popular testing frameworks, it is easy to set up, fast, has great documentation and is installed by default on CodeGrade! Read more here.
  • Selenium is an open source tool to allow web browser automation. Selenium supports automation across different browsers, platforms and programming languages and thus is the perfect tool for autograding webdev courses! Read more here.

It is now time to install both these tools in AutoTest. For Jest, CodeGrade has a wrapper script available and thus it can be very easily installed with the `cg-jest install` command. For Selenium, and all additional required packages, installing can be done using `npm` and `apt`. First we can install the required JavaScript packages: `npm install -g selenium-webdriver geckodriver`. Secondly, we will install firefox (the browser we chose to run the students’ website in) and `xvfb`, a tool to allow for virtual framebuffers as we have no real display in CodeGrade and we thus need to run in a headless web browser: `sudo apt install firefox xvfb`.

To make it easier to run and maintain this setup, we can create a simple `setup.sh` script, upload it as a fixture to CodeGrade and run that in our Global Setup Script as `$FIXTURES/setup.sh`. The full script will become (from Johan Holmberg’s GitHub):

-!- CODE language-sh -!-#!/bin/sh
# Installs Jest as per https://help.codegrade.com/user-reference/autotest-general/unit-test
cg-jest install

# Updates the Ubuntu repo settings
sudo apt update

# Installs Firefox and a headless X window manager
sudo apt install firefox xvfb

# Installs the neccessary javascript libraries
npm install -g selenium-webdriver geckodriver

# Makes the run script executable
chmod +x $FIXTURES/run.sh

Start autograding your web development courses too!

Simple Web Development HTML and CSS example assignment

For the purpose of this blog, we have designed a very simple example assignment that uses HTML, CSS and some inline JavaScript. We will use Jest and Selenium to assess two parts of the user interface of this website:

  • Clicking on one button with the ID `email` should link to a page that has the title “Email sent!”.
  • Clicking on another button with the ID `help` should open the CodeGrade Help Center (help.codegrade.com) in a new pop-up window.

The student should upload this `index.html` file, next to that they may upload `email.html` and `style.css`. 

Example webpage submission rendered in CodeGrade

Designing Selenium and Jest tests for our example assignment

Now that we have established our concrete programming tasks for this assignment, we can create tests to assess them. We will create our tests in the template Johan Holmberg has created, this makes it very easy to just focus on our tests. His template (find it here) starts with the following base code:

-!- CODE language-js -!-const { Builder, By, until } = require('selenium-webdriver');
require('geckodriver');

// This is the directory where the student's submission ends up.
// It can easily be changed to any URL.
const baseURL = 'file:///home/codegrade/student/';
// Change this to select another file to test.
const fileUnderTest = 'index.html';
const defaultTimeout = 10000;
let driver;
jasmine.DEFAULT_TIMEOUT_INTERVAL = 1000 * 60 * 5;

// Three simple functions to ease DOM navigation
const getElementByName = async (driver, name, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.name(name)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

const getElementById = async (driver, id, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.id(id)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

const getElementByTag = async (driver, tag, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.tagName(tag)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

// This is run before any test.
beforeAll(async () => {
    driver = await new Builder().forBrowser('firefox').build();
    // This could be done elsewhere if you want to test multiple pages
    await driver.get(baseURL + fileUnderTest);
});

// This is run when all tests are done. If this isn't run, the Firefox session
// lingers, so make sure it actually runs.
afterAll(async () => {
    await driver.quit();
}, defaultTimeout);

Most settings and helper functions can be left untouched. The most important things to change are the `fileUnderTest`, which should point to the file you want to test (in this case: the file `index.html` that is located in the `baseURL`, i.e. the student directory).

We can now start to write our tests, following Selenium’s documentation.

Autograding correct link after clicking a button

To test our first “email button”, we can create a simple Selenium test to click this, wait for the new page to load and then verify this page’s title. We use the following test suite for that:

-!- CODE language-js -!-// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Email" button.', () => {

    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Did you link to the "Email Sent!" page?', async () => {
        const submitButton = await getElementById(driver, 'email');
        await submitButton.click();
        const title = await driver.getTitle();
        expect(title).toBe('Email sent!');
    });
}, defaultTimeout);

We give this test suite a description that will show up in CodeGrade. In this case we chose 'Did you link to "Email Sent!" page after clicking button?'. In our test suite, we first use the helper function `getElementById` to find an element with the ID `email`. We then perform a `click()` action to this button and get the title of the page we are directed to. Finally, we expect the title of this page to be `Email sent!` and use this as a result of this test.

Autograding a pop up window

To test our second button, which should produce a pop up of our Help Center, we need a little more code. We start by navigating back to the main page of our submission: `index.html`. We then again find the required button using the `getElementById` function and looking for a button with the ID `help`. After clicking this, we get all `WindowHandles` (a pop up is a new window) and expect this to have a length of 2. Finally, we loop through all windows to see if one of them has `help.codegrade.com` in the URL. If that is the case, we pass the test.

-!- CODE language-js -!-// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Help" button.', () => {
    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Was the Help Center was opened in new tab?', async () => {
        await driver.get(baseURL + fileUnderTest);
        const submitButton = await getElementById(driver, 'help');
        await submitButton.click();
        const windowHandles = await driver.getAllWindowHandles();
        expect(windowHandles).toHaveLength(2);

        // You can only get the title of the currently active window, so we
        // loop through all the window handles to see if the confirmation
        // popup is open.
        let foundPopup = false;
        for (const win of windowHandles) {
            await driver.switchTo().window(win);
            const windowUrl = await driver.getCurrentUrl();
            if (windowUrl.includes('help.codegrade.com')) {
                foundPopup = true;
                break;
            }
        }
        expect(foundPopup).toBe(true);
    });
}, defaultTimeout);

We simply appended these two test suites to the initial base code to form our tests file `selenium.test.js`, which we will upload as a fixture to CodeGrade. Please note that you can stack `describe()` statements indefinitely, use these to add custom names to the certain labels of your Unit Test step in CodeGrade. In our case, we wrapped our two tests in another `describe('Testing the functionality of your buttons')`, which we can see back as a label in the CodeGrade UI (see screenshot later in this blog).

Setting up the student folder in CodeGrade

Now that we have uploaded our tests and global setup script as a fixtures (these will be available in the `$FIXTURES` folder) and have run `setup.sh` in the Global Setup Script to install and configure the virtual environment, it is now time to configure student environment for our Jest and Selenium testing.

Jest requires us to add a configuration file, named `jest.config.js`. This mandatory configuration file can be left empty if wished for. Our simple configuration file is found below and can be tweaked with the help of Jest’s configuration documentation here. We upload this `jest.config.js` file as a fixture.

-!- CODE language-js -!-module.exports = {
  // Indicates whether each individual test should be reported during the run
  verbose: true,
};

Now, all configuration that is left is moving all our fixtures to the right folders. For this we have created a little script called `setup_student.sh`, which moves our configuration file to the current student directory and moves our tests to a new directory called `tests`:

-!- CODE language-sh -!-#!/bin/sh

# Copies the Jest config file to the student's directory
cp $FIXTURES/jest.config.js .

# Copies the test files to the student's directory
mkdir tests
cp $FIXTURES/*test.js tests

We run this script to configure our student environment by running `​​$FIXTURES/setup_student.sh` in the Per-student setup script.

Running the tests

It is now time to create our actual Selenium tests in AutoTest to actually assess our students’ code. For this we can add a Unit Test step to our AutoTest category and run our Jest and Selenium tests. Because we need to run Firefox ‘headless’ (i.e. without a graphical display), we need to run our tests using the `xvbf` tool that we installed earlier. In total, our Program to run will become: `xvfb-run cg-jest run tests/selenium.test.js` (or `xvfb-run cg-jest run tests/` if we want to run all tests in the `tests` directory).

Selenium unit test output in CodeGrade (`describe()` statements are stacked to fill the different labels)

Summary and next steps

This definitely was one of our more advanced guides so far, with many different fixtures and tools. After finishing your AutoTest, you should have the following fixtures:

  • `jest.config.js`: required configuration file for Jest (see documentation);
  • `selenium.test.js`: the actual Jest and Selenium tests to run;
  • `setup.sh`: script to run in the Global Setup Script field;
  • `setup_student.sh`: script to run in the Per-student setup script field.

In this example, we look for an `index.html` file in our tests and thus could add Hand In Requirements to force our students to submit at least a file called `index.html`.

After you have gone through this initial setup, you are good to start duplicating your assignment or AutoTest to create more autograded HTML, CSS and JavaScript assignments. All you have to do is add new tests to the `selenium.test.js` file and optionally play around with different Jest settings in `jest.config.js`, other than that, all other files can be kept the same.

Furthermore, with our new AutoTest Caching feature, after running the (somewhat lengthy) initial setup once, it will be cached for all consecutive runs and will be a lot faster.

Want to learn more?

This blog was made thanks to the help of Johan Holmberg from Malmö University and his great GitHub repo in which he shares his experience with Selenium and CodeGrade. Many of the files we talked about in this blog can be found (albeit slightly altered) in his repository too.

Would you like to learn more about Web Development in CodeGrade? Please check out our guides on autograding MySQL and autograding SQLite too. More webinars, guides and resources on using CodeGrade for webdev courses will follow soon.

If you would like to learn more about using Selenium in CodeGrade, or other ways you can autograde your HTML, CSS and JavaScript assignments, please do not hesitate to reach out to us via support@codegrade.com

Continue reading

Grow your coding classroom without compromising on quality.