Handling of tutorial notebook cell outputs in v4

[VeckoTheGecko]

Context:

Another question is whether these should all be tutorials as jupyter notebooks with their output included, if we leave out the outputs (currently the case for v4 if I am not mistaken), or if some should be only Markdown.

For context, we're stripping output in v4 since the notebooks aren't finalised and it doesn't really make sense to commit results now.

I think it would be good to discuss this and list pros and cons wrt. maintenance, integration with (e.g.,) Binder, user experience (showing rendered outputs) - there are lots of options available (notebooks like before, or even having Markdown docs that can be run like they're notebooks). Not sure what the answer should be

Originally posted by @VeckoTheGecko in #2310

[reint-fischer]

In my view, the notebooks are a great format for tutorials and how-to guides, as users can learn by doing, understand where in the setup to change their code, and easily copy parts of the example to use themselves. From a learning perspective I think being able to see the output is really nice and intuitive. I think adding Markdown docs with explanation adds an important layer of documentation, and I think we can carefully think about what needs to be a how-to guide (e.g. release with delayed start) and what an explanation (grid indexing, stuck particles).

I am happy to try to list the pros and cons of the different options for the tutorials and how-to guides; which should we include?

1. Notebooks with rendered outputs
2. Notebooks without outputs but with Binder link
3. Markdown docs

Any others?

[erikvansebille]

Good reflection on the most important questions, @reint-fischer. I would add one more consideration:
4. Run CI on the tutorials and how-to guides to make sure their syntax and output are correct/as expected

[reint-fischer]

Thanks! I think another consideration is that it is easy to write a notebook, especially if you want to show any outputs, which I think is the case even for some explanation documents.

I wasn’t even attempting to list the considerations yet, because I first wanted to figure out what options we are comparing. I must say that I am not sure what other options we have, and that even writing a full page in Markdown with an explanation isn’t 100% clear to me. For example, in the Kernelloop tutorial, which I think is more of an explanation, there are many code blocks that you could write in Markdown, but it is still nice to be able to show an example.

[VeckoTheGecko]

I would add one more consideration:
4. Run CI on the tutorials and how-to guides to make sure their syntax and output are correct/as expected

I think testing is a different conversation than to this here

---

The options I see:

Old approach

Have a bunch of notebooks with committed outputs. They are run in CI (just ensuring they run top to bottom - no validation of outputs)

Possible new approaches

Many possibilities

There are three different avenues we can explore for docs:

1. Markdown docs
2. Examples (i.e., in the docstrings of functions)
3. Jupyter notebooks

Note that all of these options allow us to run testing on them to make sure any Python code contained still works (extracting code blocks, running examples, and running notebooks top to bottom) - hence my comment at the top. It just becomes an implementation detail depending on our documentation approach.

1. Markdown docs

Pros:

• Human readable without needing an editor
• We can use full Sphinx syntax (e.g., note/callout blocks) and any nice formatting supported by Pydata theme, as well as easily refer to the API reference using :ref:function_name derective-like syntax
• Can create executable cells using the ```{jupyter-execute} code blocks (see the Pydata theme code on Execution Libraries (docs page, source for that page) )

Cons:

• If a user "downloads" this, its more difficult to "run top to bottom". Not really nice for if people want to just download a notebook and go from there (same problem with Binder).

Additional considerations:
None

2. Examples

Pros:

• Nice to see code examples on a per-function level how a function is used

Cons:

• Additional maintainer burden (especially on a smaller project like this), where a link to the right place in a tutorial can serve a similar purpose

Additional considerations:
None

3. Notebooks

Pros:

• If a user downloads, they can run top to bottom
• Can open in Binder

Cons:

• Integration with the rest of the documentation website is more limited (e.g., you can't simply link to parts of the API reference)

Additional considerations:

• Do we want to strip out notebook outputs when committing notebooks? This makes git history less painful.
  • If yes, do we want to rebuild these outputs when deploying the website? (or is having the Binder option enough?)
  • If yes, would we want to commit the notebooks as notebooks, or as Jupytext? (https://jupytext.readthedocs.io/en/latest/ )

---

Summary

There is no one right zero-compromise solution, all projects handle all these things differently. Some projects go full markdown, some go full notebooks - larger projects have a mix of the two, erring on the side of markdown.

I think that if we do have notebooks, that we should strip the outputs when committing.

Beyond that, I have no strong preference here. I was discussing with @reint-fischer that an Examples section in docstrings isn't vital (a lot of larger projects have these, but that's an additional maintenence burden that can be significant) - so that leaves the question between markdown and notebooks.

What would be nice is that if we do have a mix between markdown and notebooks is that that split is also made on a on a documentation structure level in the docs (e.g., suppose we have a "Gallery" section - users could know that any of the gallery examples can be run in Binder, since behind the scenes they're stored as notebooks) . I think if we have one website section that is more theory/structure of Parcels which links quite a bit with the API reference, and we have another section that is more stand alone examples then it would make sense to do one with Markdown and the other with notebooks respectively.

Hope that somewhat makes sense!!

[reint-fischer]

Thanks @VeckoTheGecko!! I am curious about the jupyter-sphinx extension which let’s you use the ```{jupyter-execute} directive. I wasn’t familiar with it and I think if it provides us the opportunity to show outputs that would be great. I see that xarray uses it for their plotting documentation ( doc page | source ) for example, which I think looks very clean in the source code. The two main questions I have are:

1. How do you control in which environment to run the jupyter kernel?
2. How does it show up in git history when output is included, i.e. when are the cells executed.

For me the main pain points are between having a low(er) maintenance burden and a cleaner git history on the one hand, and easier writing and reading of tutorials and guides on the other. If we can strike a balance between those two I think it would be great. If we strip the notebooks of their outputs I think it would be useful to rebuild them when deploying the website, otherwise we might as well go with Markdown I think.

[VeckoTheGecko]

• How do you control in which environment to run the jupyter kernel?

Given that this doesn't seem to be a configuration option - I think it uses the same environment that is used to build the sphinx documentation.

2. How does it show up in git history when output is included, i.e. when are the cells executed.

AFAICT using {jupyter-execute} the outputs aren't included in the history - the cells are only executed as part of the documentation build process.

[reint-fischer]

AFAICT using {jupyter-execute} the outputs aren't included in the history - the cells are only executed as part of the documentation build process.

That is what it looked like to me too, very nice.

Given that this doesn't seem to be a configuration option - I think it uses the same environment that is used to build the sphinx documentation.

Would that complicate the writing of parcels examples in which we use other packages? Or can we just build the sphinx documentation in what is now the test-notebooks environment?

[VeckoTheGecko]

Would that complicate the writing of parcels examples in which we use other packages?

Yes, depending on where we use the {jupyter-execute} directive...

Or can we just build the sphinx documentation in what is now the test-notebooks environment?

or alternatively we can do this.

[erikvansebille]

I'm a bit on the fence about committing notebooks with output included. I don't worry too much about the git history (GitHub diffs can now parse Jupyter output quite nicely, but have two other considerations

1. If we commit notebooks with output included, we can always make sure the output is what we expect. We don't run the risk that a change in the dependencies somehow changes the output (for example plots), without us realising
2. If we execute the notebooks on the fly (e.g. with {jupyter-execute}), the advantage is that the output on the docs is always in agreement with what users would get if they run the notebooks themselves.

Not sure which of the two advantages I would prioritise if we would have to choose. Perhaps have a chat about it?