Running Unit and Acceptance Tests¶

In any process, obsessing about the wait times will yield greater enhancements than practically anything else, for longer than you might think. Automation, simplification, etc. are implementation details of that obsession. — Dan North

Unit Tests¶

Go¶

To run all unit tests, type make test in a terminal inside the development environment.

The following make targets are currently defined for tests:

test: Executes all Go (/pkg, /internal, and the internal packages of the editions), JavaScript (frontend/tests), and shell script unit tests on SQLite, with a 20 min timeout per Go package
test-go, test-js: Run the Go or JavaScript unit tests on their own
test-integration: Same as test-go, plus the integration build tag for the Insta360 stacking, import, and reconciliation matrices
test-short: Executes only fast Go tests with a 5 min timeout
test-race: Same as test-go but with the race condition detector enabled (much slower) and a 60 min timeout
test-coverage: Same as test-go but creates a coverage.txt file and a human-readable coverage.html report; timeout is elevated to 30 min
test-mariadb: Runs the Go tests against a MariaDB instance instead of SQLite
test-postgres: Runs the Go tests against the optional PostgreSQL service, which must be started first with docker compose --profile postgres up -d; it first runs reset-postgres-testdb, which drops the testdb_* databases left by earlier runs
reset-testdb: Deletes the SQLite test database files and drops the testdb_* MariaDB databases the Go tests create; test-mariadb runs reset-mariadb-testdb automatically, so call it by hand after an interrupted or targeted run
Subset targets for focused runs: test-pkg, test-api, test-ai, test-entity, test-commands, test-photoprism, test-hub, test-video

The Go targets build with the tags in GOTEST_TAGS, which defaults to slow,develop. Override it on the make command line, for example make test-mariadb GOTEST_TAGS=slow,develop,integration; an exported environment variable is ignored.

You can run single tests via go test -run in a package directory, e.g. /internal/photoprism:

$ go test -run NameOfTest

See docs for more info.

ONNX Runtime¶

The bundled face detection and embedding models run on the ONNX Runtime, so tests that exercise face detection, face embeddings, or the configuration that resolves them need the shared library to be present. The development container ships it, which is the main reason to run the Go tests inside the container rather than on the host: without it those models cannot be loaded, and the affected assertions fail in a way that reads like a code regression.

The library is loaded at startup instead of being linked into the binary, and the bindings ask for the exact C API version of the headers they vendor. An older runtime therefore fails to load rather than running with reduced functionality, and reports that the requested API version is not available. A stale development image is the usual cause, so rebuild or re-pull it before looking for the fault in the code. Outside Docker, scripts/dist/install-onnx.sh installs a matching version.

Test Frameworks¶

Go comes with a cool testing framework, it allows you to write test code using the same language, without needing to learn any library or test engine. Go advanced testing tips & tricks contains a lot of useful information. We only import testify/assert to save a few lines for common assertions.

Todo: Use a SQL mock driver to test database interactions, for example DATA-DOG/go-sqlmock.

Slow Tests¶

Slow tests and benchmarks can be skipped using the -short flag:

func TestTimeConsuming(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping test in short mode.")
    }
    ...
}

To execute:

go test -short

JavaScript¶

To run all JavaScript unit tests, type make test-js in a terminal.

Alternatively, from the frontend directory, run npm run test. Both set TZ=UTC and NODE_ENV for Vitest, so a bare npx vitest run reports date and component test failures that do not occur otherwise.

In case you want to run a single test, add .only to the test you want to run e.g.:

import { describe, it, expect } from 'vitest';

describe('Album', () => {
  it.only('should get album id', () => {
    const album = new Album({ AlbumUUID: 66 });
    expect(album.getId()).toBe('66');
  });
});

You can also run a single file or filter by test name:

cd frontend && npm run test -- tests/vitest/common/util.test.js
cd frontend && npm run test -- -t "should get album id"

For watch mode:

make vitest-watch

Available Make targets for JavaScript tests:

make test-js: Run all JavaScript unit tests
make vitest-watch: Run Vitest in watch mode
make vitest-coverage: Run Vitest with coverage report
make vitest-component: Run component tests

Test coverage output is saved to frontend/coverage/index.html.

Test Frameworks¶

To test JavaScript code we use Vitest together with Vite, jsdom, Vue Test Utils, and @vitest/coverage-v8.

Acceptance Tests¶

Before you proceed run

make dep-acceptance

Running Tests Inside the Docker Container¶

In the development container environment, you can run the tests in headless chrome:

make acceptance-run-chromium

To run the acceptance tests on MariaDB or PostgreSQL instead of SQLite, pass the database on the make command line, e.g. make acceptance-run-chromium DB=mariadb or DB=postgres (requires the postgres profile); a DB environment variable is ignored. Add TC_USE_VIDEO=1 to record videos of failed tests in frontend/tests/acceptance/videos/.

Test Frameworks¶

Our goal was to implement UI acceptance tests using JavaScript, so that frontend developers are able to run and write them without learning Go.

To make a final decision, we compared TestCafe, Cypress and Nightwatch.js. We agreed on using TestCafe as tests were the most stable and pretty fast (because no long timeouts are needed).

Feature	TestCafe	Cypress	Nightwatch.js
Supported Browsers Local	Chrome, Firefox, Opera, Safari, Internet Explorer, Microsoft Edge, Google Chrome Canary, Chromium	Chrome, Chromium, Chrome Canary, Electron	Geckodriver, Chromedriver, Microsoft Webdriver, Safaridriver
Supported Browsers Headless	Chrome, Firefox	Electron	Problems running headless
Continuous Integration Support	yes	yes	yes
Setup	easy via npm	easy via npm	easy via npm
Usability	+++	+++	++
Speed (3 tests)	2 min (headless chrome and firefox) 1 min (only chrome headless) 1 min (chrome headed)	5 min (headless electron) 2,5 min (chrome headed)	7 min (chrome headed) headless not working
Stability	nice	unstable --> waiting times needed	unstable --> waiting times needed
Documentation	+++	++	++
Notes	easy to find elements	easy to find elements	additional library needed to find selectors by text

Other test libraries and frameworks we currently don't use: