Testability and "Wait, it broke again?"

[Salanto]

Testability is the hallmark of great software. Given we use Github Actions to get quick feedback on build errors on all platforms, it is only natural that we should consider the extra mile and start rewriting code with testability in mind.

Sadly, after reading through the major classes, it becomes obvious that AO2-Client has become so tightly woven that many features are not testable without a lot of overhead and hackery to get QTest to be able to properly test each individual segment, meaning we would spend a lot of time writing tests that may break simply due to some minor aspect changing and without an easy way to properly locate what exactly broke.

Moving forward contributions should be evaluated on grounds of testability as well to ensure that new code can be verified using a testing library of our choice to stop the guessing game of what broke this random feature today.

[oldmud0]

I have never worked in the game industry, and game testing is something that I'd really like to know more about. My current knowledge of game testing is that it's relatively hard to automate testing for anything other than pure functions, at least without a custom library specifically for testing the game. There's also automated screenshot testing (like Dolphin's testing pipeline), which compares the game pixel-by-pixel at certain key moments. But most testing is relegated to human QA testers who deal with the brunt of testing objectives.

It would probably be beneficial to start with a black-box test for critical paths - start a server, have the client connect, click a character button on the grid, try to write/send evidence, etc. No need to expose the courtroom.cpp mess - just as long as all of the widgets are well-named, seems like that's all that's needed for testing.

[Salanto]

A blackbox approach can work, but there are some questions left unanswered :

If we were to join a server, we would either need a way to host it in an action or someone needs to perma-host one, which in turn would also cause it to be a target for annoying people mischievous individuals. Any ideas on that cause I doubt actions would be happy if we tried to spawn an Akashi Server inside the action and connect our client to it.

A white box approach would help us test segments, but as your fairly pointed out there are some (almost all) segments that are not testable on an individual scale.

[oldmud0]

Any ideas on that cause I doubt actions would be happy if we tried to spawn an Akashi Server inside the action and connect our client to it.

This should work perfectly fine. Actions instances should allow listening on the loopback interface. Heck, I'm pretty sure you can run containers (Docker, Podman) inside Actions and they will work just fine.

[Salanto]

After some research if found this article on Github about service containers : https://docs.github.com/en/actions/using-containerized-services/about-service-containers

If we containerise a test AO2-Server we should be able to do black-box testing on connectivity and handshake behaviour, etc without needing a fully-fledged external AO2 server.