Make an API online documentation

[kelson42]

via readthedocs.com

[elfkuzco]

@kelson42 , I would like to work on this. I don't have much experience with building the docs with readthedocs.com but I am aware you could either use Sphinx or MkDocs as the documentation generators. Do you have any generator in mind? I reckon I could get it done in a week or less if it is assigned to me.

[benoit74]

We need something similar to https://github.com/openzim/python-scraperlib/, which is based on mkdocs.

For this repo, most of the work was done with openzim/python-scraperlib#215 and openzim/python-scraperlib#225, and openzim/python-scraperlib@22ddefd

Note that this repo is only a very thin wrapper on top of the libzim, so the purpose is mainly to expose / document all available API(s), and explain this fact (that it's a thin wrapper ...).

Your work will mostly consist in setting up everything in this repo, I've just setup everything needed on readthedocs.org side.

[elfkuzco]

Okay. I will work on it.

[elfkuzco]

I have set up something that works locally. However, there are a few hiccups and I would like to know how best to proceed.
The main issue is that the mkdocstring plugin doesn't parse docstrings in .pyx files. According to the conversations in this issue, it works for compiled code (in this case libzim.so). I didn't try this though as it could mean compiling it on readthedocs or ci/cd.

On the other hand, it generates the docs for the .pyi files which is shown in the image attached below. Note that I added some docstrings in the .pyi files, otherwise it showed only class names, methods and attrs. I only did it to check things worked as I expected.

Obviously, adding the docstring in all the .pyi files isn't feasible as it would mean duplication of the docstrings in the .pyx files.

So, should I stick to just working with the bare enumeration (no docstrings) of the class, method and attribute names that the .pyi files expose? Or should I explore the path of extracting the docstring from the compiled libzim.so file? What do you think?

[rgaudin]

If possible, I'd like to see how it works via the compiled lib ; just so we have all the information at hand to decide.

[elfkuzco]

If possible, I'd like to see how it works via the compiled lib ; just so we have all the information at hand to decide.

I finally set it up to work with the compiled lib but it's not quite informative.

It just lists __name__ and __package__ as methods which has the same content as this comment.

I tried to use their internal tool griffe to inspect the module and see it's AST tree and it shows a tree which contains just those two attributes. I don't really know much about all the options involved with using setup.py (perhaps to add more data to the compiled binary) and it might take longer to figure out.

[rgaudin]

Thanks ; I'd like to take a quick look myself before we decide to move the dosctrings to pyi which would be suboptimal (docstrings not next to code are doomed to become incorrect overtime)

[rgaudin]

OK, I've looked at it and I think we can make it work

I couldn't make it work properly though and had to hack the Python handler.

The thing is that griffe does find the proper docstrings when you pass it the -x (force inspection) flag.

❯ griffe -x libzim

The problem being that griffe is called by the python handler of mkdocstrings but never passes that information.

Here's my fix (toggling it on a new option):

*** mkdocstrings_handlers/python/handler.py.orig	2025-01-22 07:53:13
--- mkdocstrings_handlers/python/handler.py	2025-01-22 07:53:18
*************** class PythonHandler(BaseHandler):
*** 318,323 ****
--- 318,324 ----
                  modules_collection=self._modules_collection,
                  lines_collection=self._lines_collection,
                  allow_inspection=final_config["allow_inspection"],
+                 force_inspection=final_config["force_inspection"],
              )
              try:
                  for pre_loaded_module in final_config.get("preload_modules") or []:

There are ways to extend and customize with extension, plugins, etc but I find the reasonable approach is for you to fork mkdocstrings and work off your fork. Please submit them a PR with the fix, I believe it can make it and if it can't, they'll probably tell you how to do it otherwise.

Now it's not perfect and I've noted a bunch of other things to fix:

• There are typos in docstrings
• We use use a mix of dosctring styles. I believe we want to harmonize towards google. I fixed get_entry_by_path() in that sense.
• @property decorated functions appear as class instead of the property's return type. that's the main issue I've found. Hopefully there's a way to fix that as well but since it's missing from the griffe output I'm not sure. I've cheated in Archive.filename so the return type is visible. If you can't fix it, please apply the same workaround and maybe open a ticket at griffe for that.
• document (inline) all options of the mkdocs.yaml file.
• that file also needs to be cleaned up as it's a collage of griffe's own and scraperlib's.

⚠️ you need to rebuild the wrapper (SIGN_APPLE=0 hatch run build-ext) anytime you change docstrings for it to be found by mkdocs (since it's use inspection and not reading source code). That sounds obvious but let's say I've realized that a bit late.

What I did is in the docs branch here.

[elfkuzco]

Thank you very much @rgaudin