Ensuring Validity of Code Snippets in Documentation

[hunhoffe]

I am researching options to keep the programming_guide code snippets tested and updated. I was hoping to start a discussion on possible solutions.

Problem

Right now, code snippets are not tested for correctness by the CI which may lead to code snippets silently becoming outdated. It's possible that copied snippets and source code files could also become out of sync, since the snippets require a person to manually copy code from the source into the markdown file.

Possible Solutions

1. Jupyter Notebooks

Converting examples to Jupyter Notebooks might be a way forward, and then those notebooks could be executed as part of the CI process to ensure they remain up-to-date.

• Pros: Jupyter Notebooks are easy to modify in self-guided experiments and could be a useful teaching tool
• Cons: Jupyter Notebooks would require additional environment setup instructions, may remove examples of how to make a simple stand-alone program from the programming-guide if used too heavily, and would require an unknown amount of effort to convert current examples to notebook format.

2. Markdown Autodocs Action

I found a GitHub action called Markdown Autodocs that might also be a path forward. Markdown Autodocs scans selected markdown files for specifically formatted comments, and then inserts code snippets into those markdown files and creates a commit to the repo with the update. I created a proof-of-concept repo/action using markdown autodocs here. Specifically, the code snippet managed by markdown autodocs here. For what it's worth, I found it pretty easy to use.

• Pros: Not many modifications need to be made to get the benefits of auto-populated code snippets, code can be removed from markdown files and places in python/mlir files which may be tested as normal with lit tests.
• Cons: The markdown autodocs action needs write access to the repo (could be a concern depending on AMDs policies on third-party github actions), auto-commits can clutter commit history a bit

Other options??

Perhaps something I haven't thought of?

[stephenneuendorffer]

My experience using things like markdown-autodocs is that they are very fragile. (perhaps more fragile than the thing they are trying to solve) So, it solves the "don't have something in 2 places" problem, but not really the "when did this thing change enough that someone needs to look at the documentation again." problem. Jupyter notebooks also have their own set of problems, since they mix an intention ("Run this code") with the result of running the code in a single file. Personally, I prefer to focus on well commented examples (which can be tested) and writing documentation which is more conceptual, so it doesn't change significantly as the code changes.

[hunhoffe]

To paraphrase a little (and let me know if my understanding is wrong/incomplete), an Option 3 would be to remove all/most code snippets from the programming guide entirely.

• Pros: Simplicity and maintainability
• Cons (as I see them): Could be a bumpy ride for newcomers as they try to bridge theoretical understanding with concrete examples, since documentation on writing code may not exist other than in comments; removing code snippets would require a fair amount of restructuring to the programming guide, especially sections 1 and 2, since they rely heavily on code snippets.