Autograding web user interfaces with Selenium and Jest in CodeGrade

Simple Web Development HTML and CSS example assignment

For the purpose of this blog, we have designed a very simple example assignment that uses HTML, CSS and some inline JavaScript. We will use Jest and Selenium to assess two parts of the user interface of this website:

• Clicking on one button with the ID `email` should link to a page that has the title “Email sent!”.
• Clicking on another button with the ID `help` should open the CodeGrade Help Center (help.codegrade.com) in a new pop-up window.
  

The student should upload this `index.html` file, next to that they may upload `email.html` and `style.css`. 

Designing Selenium and Jest tests for our example assignment

Now that we have established our concrete programming tasks for this assignment, we can create tests to assess them. We will create our tests in the template Johan Holmberg has created, this makes it very easy to just focus on our tests. His template (find it here) starts with the following base code:

-!- CODE language-js -!-const { Builder, By, until } = require('selenium-webdriver');
require('geckodriver');

// This is the directory where the student's submission ends up.
// It can easily be changed to any URL.
const baseURL = 'file:///home/codegrade/student/';
// Change this to select another file to test.
const fileUnderTest = 'index.html';
const defaultTimeout = 10000;
let driver;
jasmine.DEFAULT_TIMEOUT_INTERVAL = 1000 * 60 * 5;

// Three simple functions to ease DOM navigation
const getElementByName = async (driver, name, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.name(name)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

const getElementById = async (driver, id, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.id(id)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

const getElementByTag = async (driver, tag, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.tagName(tag)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};

// This is run before any test.
beforeAll(async () => {
    driver = await new Builder().forBrowser('firefox').build();
    // This could be done elsewhere if you want to test multiple pages
    await driver.get(baseURL + fileUnderTest);
});

// This is run when all tests are done. If this isn't run, the Firefox session
// lingers, so make sure it actually runs.
afterAll(async () => {
    await driver.quit();
}, defaultTimeout);

Most settings and helper functions can be left untouched. The most important things to change are the `fileUnderTest`, which should point to the file you want to test (in this case: the file `index.html` that is located in the `baseURL`, i.e. the student directory).

We can now start to write our tests, following Selenium’s documentation.

Autograding correct link after clicking a button

To test our first “email button”, we can create a simple Selenium test to click this, wait for the new page to load and then verify this page’s title. We use the following test suite for that:

-!- CODE language-js -!-// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Email" button.', () => {
‍
    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Did you link to the "Email Sent!" page?', async () => {
        const submitButton = await getElementById(driver, 'email');
        await submitButton.click();
        const title = await driver.getTitle();
        expect(title).toBe('Email sent!');
    });
}, defaultTimeout);

We give this test suite a description that will show up in CodeGrade. In this case we chose 'Did you link to "Email Sent!" page after clicking button?'. In our test suite, we first use the helper function `getElementById` to find an element with the ID `email`. We then perform a `click()` action to this button and get the title of the page we are directed to. Finally, we expect the title of this page to be `Email sent!` and use this as a result of this test.

Autograding a pop up window

To test our second button, which should produce a pop up of our Help Center, we need a little more code. We start by navigating back to the main page of our submission: `index.html`. We then again find the required button using the `getElementById` function and looking for a button with the ID `help`. After clicking this, we get all `WindowHandles` (a pop up is a new window) and expect this to have a length of 2. Finally, we loop through all windows to see if one of them has `help.codegrade.com` in the URL. If that is the case, we pass the test.

-!- CODE language-js -!-// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Help" button.', () => {
    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Was the Help Center was opened in new tab?', async () => {
        await driver.get(baseURL + fileUnderTest);
        const submitButton = await getElementById(driver, 'help');
        await submitButton.click();
        const windowHandles = await driver.getAllWindowHandles();
        expect(windowHandles).toHaveLength(2);

        // You can only get the title of the currently active window, so we
        // loop through all the window handles to see if the confirmation
        // popup is open.
        let foundPopup = false;
        for (const win of windowHandles) {
            await driver.switchTo().window(win);
            const windowUrl = await driver.getCurrentUrl();
            if (windowUrl.includes('help.codegrade.com')) {
                foundPopup = true;
                break;
            }
        }
        expect(foundPopup).toBe(true);
    });
}, defaultTimeout);

We simply appended these two test suites to the initial base code to form our tests file `selenium.test.js`, which we will upload as a fixture to CodeGrade. Please note that you can stack `describe()` statements indefinitely, use these to add custom names to the certain labels of your Unit Test step in CodeGrade. In our case, we wrapped our two tests in another `describe('Testing the functionality of your buttons')`, which we can see back as a label in the CodeGrade UI (see screenshot later in this blog).

Setting up the student folder in CodeGrade

Now that we have uploaded our tests and global setup script as a fixtures (these will be available in the `$FIXTURES` folder) and have run `setup.sh` in the Global Setup Script to install and configure the virtual environment, it is now time to configure student environment for our Jest and Selenium testing.

Jest requires us to add a configuration file, named `jest.config.js`. This mandatory configuration file can be left empty if wished for. Our simple configuration file is found below and can be tweaked with the help of Jest’s configuration documentation here. We upload this `jest.config.js` file as a fixture.

-!- CODE language-js -!-module.exports = {
  // Indicates whether each individual test should be reported during the run
  verbose: true,
};

Now, all configuration that is left is moving all our fixtures to the right folders. For this we have created a little script called `setup_student.sh`, which moves our configuration file to the current student directory and moves our tests to a new directory called `tests`:

-!- CODE language-sh -!-#!/bin/sh

# Copies the Jest config file to the student's directory
cp $FIXTURES/jest.config.js .

# Copies the test files to the student's directory
mkdir tests
cp $FIXTURES/*test.js tests

We run this script to configure our student environment by running `​​$FIXTURES/setup_student.sh` in the Per-student setup script.

Running the tests

It is now time to create our actual Selenium tests in AutoTest to actually assess our students’ code. For this we can add a Unit Test step to our AutoTest category and run our Jest and Selenium tests. Because we need to run Firefox ‘headless’ (i.e. without a graphical display), we need to run our tests using the `xvbf` tool that we installed earlier. In total, our Program to run will become: `xvfb-run cg-jest run tests/selenium.test.js` (or `xvfb-run cg-jest run tests/` if we want to run all tests in the `tests` directory).

Summary and next steps

This definitely was one of our more advanced guides so far, with many different fixtures and tools. After finishing your AutoTest, you should have the following fixtures:

• `jest.config.js`: required configuration file for Jest (see documentation);
• `selenium.test.js`: the actual Jest and Selenium tests to run;
• `setup.sh`: script to run in the Global Setup Script field;
• `setup_student.sh`: script to run in the Per-student setup script field.
  

In this example, we look for an `index.html` file in our tests and thus could add Hand In Requirements to force our students to submit at least a file called `index.html`.

After you have gone through this initial setup, you are good to start duplicating your assignment or AutoTest to create more autograded HTML, CSS and JavaScript assignments. All you have to do is add new tests to the `selenium.test.js` file and optionally play around with different Jest settings in `jest.config.js`, other than that, all other files can be kept the same.

Furthermore, with our new AutoTest Caching feature, after running the (somewhat lengthy) initial setup once, it will be cached for all consecutive runs and will be a lot faster.

Want to learn more?

This blog was made thanks to the help of Johan Holmberg from Malmö University and his great GitHub repo in which he shares his experience with Selenium and CodeGrade. Many of the files we talked about in this blog can be found (albeit slightly altered) in his repository too.

Would you like to learn more about Web Development in CodeGrade? Please check out our guides on autograding MySQL and autograding SQLite too. More webinars, guides and resources on using CodeGrade for webdev courses will follow soon.

If you would like to learn more about using Selenium in CodeGrade, or other ways you can autograde your HTML, CSS and JavaScript assignments, please do not hesitate to reach out to us via support@codegrade.com.