Functional Testing

[chrisdenman]

Hi,

I've got some PRs that improve bits of the HTML code (amongst some purely content changes too).

The changes (and PRs) contain very few commits so they are not hard to test.

However, I'm a little (5%?) nervous about altering the HTML since I don't have much test coverage on browsers other than the one I use here (firefox/chrome). This is all manual testing.

I've been thinking about writing some automated tests for the project, these would most likely take the form of automated Selenium tests. This allow tests to be written that control real browsers (on desktop machines or mobile devices). The tests would load pages (on production or local machines) and then check the content is what is expected.

This testing / QA strategy would provide some confidence that changes do not introduce problems into the rendering of the book, before branches are merged. It would boost the confidence that the book works across a range of devices.

This work could be incorporated into the main codebase (maybe git modules (I'll need to research that)) or, as a separate project.

We could start by simply running some tests against the production site which could take screenshots and then compare these (manually or with a more advanced strategy), against proposed branches, to get a baseline of what the site should like like.

I notice that there's a PR to generate the site from markdown. I feel that with automated testing in place, it would greatly ease this transition as we could be confident that no regressions have been introduced. Also, if we wanted to make device specific versions of the site (mobile e.g.) this would help there too.

My expertise is in the JVM (Java/Kotlin/Groovy) realm, so one of those would probably be my choice of language with which to code the project. However, I'm open to other ideas if other people have other skills and are willing and likely to contribute.

I would greatly appreciate your thoughts. If anyone wants to get involved / has experience in coding / whatever... please share your thoughts/comments/ideas etc. any input would be greatly received.

Thanks!

notes to self 

1. see if GitHub Actions can help with automating the functional testing before branch merging
2. assess at the state of the art with respect to automated browser driving these days

[chrisdenman]

Hi!

A quick update. I've created a project chrisdenman/test.learnyouahaskell.github.io for functionally testing the book as above.

In terms of functional tests it's pretty bare-boned currently. There are tests that cover:

• The Landing Page - ensuring the links are correct (text displayed, links)
• The Chapters Page - ensuring that the links are correct (text displayed, link to correct page + fragment)

, also, they produce screenshots of every page in the site. So that when updating the site we/you can be confident that there's been no regressions.

The current project enables you to specify (via config) a owner/branch to check out. It then runs all the tests and takes all the screenshots. There's not much in the way of documentation at the moment but, that's next on my list.

Ta