Running Unit and Acceptance Tests¶

In any process, obsessing about the wait times will yield greater enhancements than practically anything else, for longer than you might think. Automation, simplification, etc. are implementation details of that obsession. — Dan North

Unit Tests¶

Go¶

To run all unit tests, type make test or go test ./internal/... in a terminal.

These make targets are currently defined for tests:

test: Executes all tests found in a) /internal with a timeout of 20 min and verbose output and b) frontend/tests/unit
test-short: Executes only fast tests found in /internal with a timeout of 5 min and verbose output
test-race: Same as test but with race condition detector (much slower) and higher timeout of 60 min
test-codecov: Same as test but creates a coverage log in coverage.txt and sends it to Codecov (don't use it locally)
test-coverage: Same as test but creates a coverage.txt file as well as a human-readable report in coverage.html; timeout is elevated to 30 min

You can run single tests via go test -run in a package directory, e.g. /internal/photoprism:

$ go test -run NameOfTest

See docs for more info.

Test Frameworks¶

Go comes with a cool testing framework, it allows you to write test code using the same language, without needing to learn any library or test engine. Go advanced testing tips & tricks contains a lot of useful information. We only import testify/assert to save a few lines for common assertions.

Todo: Use a SQL mock driver to test database interactions, for example DATA-DOG/go-sqlmock.

Slow Tests¶

Slow tests and benchmarks can be skipped using the -short flag:

func TestTimeConsuming(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping test in short mode.")
    }
    ...
}

To execute:

go test -short

Javascript¶

To run all javascript unit tests, type make test-js in a terminal.

In case you want to run a single test add .only to the test you want to run e.g.:

 it.only("should get album id",  () => {
        const values = {
            ID: 5, AlbumName: "Christmas 2019", 
            AlbumSlug: "christmas-2019", AlbumUUID: 66
        };
        const album = new Album(values);
        const result = album.getId();
        assert.equal(result, "66");
    });

Test coverage output is saved to frontend/coverage/html

Test Frameworks¶

To test javascript code we use mocha in combination with karma, chai, sinon and the karma-istanbul-coverage-reporter.

Acceptance Tests¶

Before you proceed run

make dep-acceptance

Running Tests Inside the Docker Container¶

In the development container environment, you can run the tests in headless chrome:

make acceptance-run-chromium

Test Frameworks¶

Our goal was to implement UI acceptance tests using JavaScript, so that frontend developers are able to run and write them without learning Go.

To make a final decision, we compared TestCafe, Cypress and Nightwatch.js. We agreed on using TestCafe as tests were the most stable and pretty fast (because no long timeouts are needed).

Feature	TestCafe	Cypress	Nightwatch.js
Supported Browsers Local	Chrome, Firefox, Opera, Safari, Internet Explorer, Microsoft Edge, Google Chrome Canary, Chromium	Chrome, Chromium, Chrome Canary, Electron	Geckodriver, Chromedriver, Microsoft Webdriver, Safaridriver
Supported Browsers Headless	Chrome, Firefox	Electron	Problems running headless
Continuous Integration Support	yes	yes	yes
Setup	easy via npm	easy via npm	easy via npm
Usability	+++	+++	++
Speed (3 tests)	2 min (headless chrome and firefox) 1 min (only chrome headless) 1 min (chrome headed)	5 min (headless electron) 2,5 min (chrome headed)	7 min (chrome headed) headless not working
Stability	nice	unstable --> waiting times needed	unstable --> waiting times needed
Documentation	+++	++	++
Notes	easy to find elements	easy to find elements	additional library needed to find selectors by text

Other test libraries and frameworks we currently don't use: