automate running example notebooks with github actions

[alxndrkalinin]

We have a set of example notebooks that can be treated as end-to-end tests on real data and could possibly be automated using github actions.

Rough plan would be:

• update notebooks to output their results, which we then can compare with what we know is correct
• figure out how to automate running notebooks with github actions (e.g. this one)

[afermg]

From my notes, I usually use jupytext to automate running scripts into notebooks and then a website (I put it in a bash script and run from a workflow via nix for reproducibility, but the last part is far from crucial), but another couple of tools that may be more useful in this case:

• doctest: test documentation in the dostrings
• testbook: Test notebooks directly (haven't used it myself though)

I'm sure @gnodar01 has faced this issue and could have some insight as to which tools are best.

[gnodar01]

There are parts of nbdev that will let you do this pretty easily.

Nbdev is modular, so you don't have to use all their machinery (e.g. you might not want exporting cells to scripts, creating quarto outputs, doc generation, commit hooks, packaging, directives), but rather the parts relevant to you (tests, gh actions workflows, and the fast + parameterized notebook runner).

The run-notebook gh action @alxndrkalinin linked achieves roughly the same thing, so not having tried it, I can't give a definitive answer on which approach would be better. At the very least, nbdev is much better maintained. That papermill fork hasn't been updated in 6 years. The action itself in 3 years.

Although papermill itself does seem good, and is part of the same org as the testbook that @afermg linked.