Testing the untested parts of TiddlyWiki

[hoelzro]

For the last two changes I've made to TiddlyWiki, I've wanted to add tests, but they've been in two parts of the code (namely, the plugin-loading part of the boot process and the menubar plugin) that don't seem to have testing infrastructure set up.

At @Jermolene's suggestion, I'd like to start a discussion about testing TiddlyWiki, specifically:

Which parts are and are not covered by tests?

As far as "covered" goes, widgets, parsing, and filters seem to be tested.

The boot process and plugins are my own immediate examples of things that are not covered, but I'd like to identify others.

How do we test these parts?

For the plugin-loading code, @Jermolene suggested a refactor, but undertaking a refactor without tests makes me nervous, so I have a bit of a dilemma 😅

I was thinking that maybe some sort of roundtrip/end-to-end testing could be done to provide some assurance during a refactor - the idea here would be to build entire wikis for test cases, and leverage Playwright to check for regressions. These would be slow, but as chunks of functionality get covered by more unit-style tests after some refactoring, the Playwright-based tests could be deleted.

[pmario]

... the Playwright-based tests could be deleted.

Tests should never be deleted. They make sure that we catch regressions and they need to be executed every time a PR is requested.

It seems we already are "scratching" the netlify CI time limits.

The slowest and probably memory consuming elements is creating the TW static html pages, which are also created every time a PR is requested.

[hoelzro]

I should clarify - when I say "the Playwright-based tests could be deleted", I mean any new Playwright-based tests meant to make sure regressions don't creep in while refactoring the boot process is happening, and only once other, faster tests are written to cover the boot process.

[pmario]

I am not really sure what you mean by boot related code. The core repo has a boot.js and a bootprefix.js. Do you mean those or do you mean when the wiki gets started.

Boot*.js does mainly instantiate the different modules by loading them from files or from tiddlers. So we do not test the boot functions but the module functions once they are initialised. If they are available boring did work.

[hoelzro]

Ok, let me provide a more concrete example - here's a PR I submitted a few weeks ago: #9025

I'd found a bug where using modified in tiddlywiki.files with a plugin wasn't resulting in the proper modified value in the plugin tiddlers - this is the kind of situation I'd like to add a test for.

What I'm proposing above would be to refactor loadTiddlersFromSpecification and related code so that it's easier to write something like a unit test for it (eg. passing in an object abstracting over filesystem operations or something) and then actually writing that unit test - but first having something more like an end-to-end test in place to catch any regressions the refactoring might introduce.

[hoelzro]

Another thing that comes to mind - I know that historically, TiddlyWiki has been hesitant to use third-party dependencies, but I was wondering if others would be open to leveraging modules like mock-fs for things like the loadTiddlersFromSpecification code.

[hoelzro]

Here's an example of what I have in mind for these kinds of tests: https://github.com/hoelzro/TiddlyWiki5/tree/example-e2e-test/full-wiki-tests/plugin-derived-fields

I'm not sure about how we'd want to structure these tests and run them in CI, and we'd probably want to use JavaScript rather than shell to actually drive tests, but the basic idea is to set up full wikis demonstrating the behavior under test and run each of their test.sh scripts. It's a bit heavy, but these tests would allow us to check for regressions as refactoring takes place.

[pmario]

Just a remark. For TW we need tests that work win Linux and Windows.