Controlled End-to-End Testing of QGIS Plugins

At Nelen & Schuurmans we build and maintain several QGIS plugins. Because we like to ship improvements and new features quickly, we try to release new versions often.

Frequent releases only work if the quality is consistently maintained. To help with that (and to reduce manual testing), we rely on automated tests. Besides unit tests, that check isolated pieces of code, we also run end-to-end tests that simulate real user workflows. Ideally, these tests closely mimic what the users do: clicking buttons in the plugin interface. From there, the test follows the whole plugin logic, interacting with external services such as API's, and eventually verifying that the expected result actually happened (for example, a file ending up in cloud storage). In other words, we try to test the full chain to make sure all components work together.

Running these kinds of tests for a GUI application like QGIS is not trivial. We wanted our tests to run both automatically in a headless CI/CD environment (such as GitHub Actions), and locally on a developer’s machine. In both cases, we would like to visually inspect failed tests.

By combining Docker and WSL / Linux X11 window system, and limited mocking of the QgisInterface, we ended up with a single setup that suits are needs. In this presentation, we’ll share how we set this up, the challenges we encountered, and the lessons we learned along the way.