Skip to content

Use sphinx-autoapi to generate API documentation #3646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
7b712cc
add autoapi gubbins to sonf.py
valeriupredoi Jun 6, 2024
2623191
first draft of readthedocs config file for autoapi
valeriupredoi Jun 6, 2024
53e81a3
actually remove the pip bit
valeriupredoi Jun 6, 2024
fe20722
correct path to source code
valeriupredoi Jun 11, 2024
55ae5a7
try with a temporary version placeholder
valeriupredoi Jun 11, 2024
1cf623c
rm autodocsumm
valeriupredoi Jun 11, 2024
4001b3f
duh correct RTD syntax for extension
valeriupredoi Jun 11, 2024
bf35f94
add independent requirements file
valeriupredoi Jun 11, 2024
9384eb9
revert extension name
valeriupredoi Jun 11, 2024
26e6edb
call the autoapi module installation from a separate requirements file
valeriupredoi Jun 11, 2024
1b17448
add theme dep to requirements
valeriupredoi Jun 11, 2024
9e41e1b
throw some configuration at autoapi
valeriupredoi Jun 11, 2024
3cb33ef
exclude diag_scripts
valeriupredoi Jun 11, 2024
9324b25
add only select api elements to be documented
valeriupredoi Jun 11, 2024
bd12331
comment out all autodoc settings
valeriupredoi Jun 11, 2024
09b81ad
turn off generation of index.rst and set the autoapi root dir
valeriupredoi Jun 11, 2024
5b7d23d
remove that silly index.rst already!
valeriupredoi Jun 11, 2024
5eb2186
maybe we want that index.rst after all
valeriupredoi Jun 11, 2024
8a2dfa2
try shove a sys appended path for autodoc
valeriupredoi Jun 11, 2024
e75d0c3
try shove a sys appended path for autodoc
valeriupredoi Jun 11, 2024
b22ca27
fix that terrible previous commit
valeriupredoi Jun 11, 2024
f07c760
Disable autodoc and select only wanted submodules with autoapi
bouweandela Jun 12, 2024
c546796
remove autodocsumm and pydata sphinx theme dependencies
valeriupredoi Jun 12, 2024
9901882
remove autodocsumm and pydata sphinx theme dependencies
valeriupredoi Jun 12, 2024
0c0059f
add ignore to autoapi congig
valeriupredoi Jun 12, 2024
c3d14fe
correct wildcard
valeriupredoi Jun 12, 2024
3ea9878
Add __init__.py files and add __all__
bouweandela Jun 12, 2024
06b55b9
Merge branch 'use_sphinx-autoapi' of https://github.com/ESMValGroup/E…
valeriupredoi Jun 12, 2024
01fd8fc
add testkw to the ignored list
valeriupredoi Jun 12, 2024
1d2ed6b
nevermind, am stoopid
valeriupredoi Jun 12, 2024
9f18b66
add names.py to ignores
valeriupredoi Jun 12, 2024
9f85e78
reinstate names.py
valeriupredoi Jun 12, 2024
b49b34c
comment out import in sraed init
valeriupredoi Jun 12, 2024
a0584e8
comment out all imports from shared init
valeriupredoi Jun 12, 2024
e791641
reinstate some imports
valeriupredoi Jun 12, 2024
d6fce5b
Comment out ignores
bouweandela Jun 19, 2024
6392e27
Fix formatting
bouweandela Jun 19, 2024
024896a
Fix some warnings
bouweandela Jun 19, 2024
9719bca
Fix docs build
bouweandela Jun 19, 2024
f9a568f
make package level imports
valeriupredoi Jun 20, 2024
53e7f91
make package level imports
valeriupredoi Jun 20, 2024
9c57a60
make package level imports
valeriupredoi Jun 20, 2024
c6c3a3a
make package level imports
valeriupredoi Jun 20, 2024
0d4f069
make package level imports
valeriupredoi Jun 20, 2024
eea6b00
make package level imports
valeriupredoi Jun 20, 2024
a9edecb
make package level imports
valeriupredoi Jun 20, 2024
5493f41
make package level imports
valeriupredoi Jun 20, 2024
ae53507
Merge branch 'main' into use_sphinx-autoapi
valeriupredoi Jun 20, 2024
a14d7e1
Merge branch 'main' into use_sphinx-autoapi
valeriupredoi Jun 25, 2024
08a5eba
correct for souble import
valeriupredoi Jun 25, 2024
5dc083e
correct for souble import
valeriupredoi Jun 25, 2024
7408d99
couple more conflict fixing bloopers
valeriupredoi Jun 25, 2024
ac86b9e
Merge branch 'main' into use_sphinx-autoapi
valeriupredoi Jul 9, 2024
ccfde85
Merge branch 'main' into use_sphinx-autoapi
valeriupredoi Jul 9, 2024
c228572
Merge branch 'main' into use_sphinx-autoapi
valeriupredoi Sep 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 6 additions & 18 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,7 @@ version: 2
build:
os: ubuntu-22.04
tools:
# updated and deployed from Aug 1, 2023
python: "mambaforge-22.9"
jobs:
pre_create_environment:
# update mamba just in case
- mamba update --yes --quiet --name=base mamba 'zstd=1.5.2'
- mamba --version
- mamba list --name=base
post_create_environment:
- conda run -n ${CONDA_DEFAULT_ENV} mamba list
# use conda run executable wrapper to have all env variables
- conda run -n ${CONDA_DEFAULT_ENV} mamba --version
- conda run -n ${CONDA_DEFAULT_ENV} pip install . --no-deps

# Declare the requirements required to build your docs
conda:
environment:
environment.yml
python: "3.12"

# Build documentation in the doc directory with Sphinx
sphinx:
Expand All @@ -36,3 +19,8 @@ sphinx:
# If using Sphinx, optionally build your docs in additional formats such as PDF
formats:
- pdf

# Install auotoapi requirement
python:
install:
- requirements: doc/sphinx/source/requirements.txt
18 changes: 14 additions & 4 deletions doc/sphinx/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@
root = Path(__file__).absolute().parent.parent.parent.parent
sys.path.insert(0, str(root))

from esmvaltool import __version__
# from esmvaltool import __version__
__version__ = "2.10.0"

# -- RTD configuration ------------------------------------------------

Expand Down Expand Up @@ -61,7 +62,7 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
# 'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.extlinks',
'sphinx.ext.intersphinx',
Expand All @@ -71,9 +72,18 @@
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode',
'sphinx.ext.napoleon',
'autodocsumm',
'autoapi.extension',
]

# Autoapi configuration
autoapi_dirs = ['../../../esmvaltool']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible change this list so only these are documented? https://github.com/ESMValGroup/ESMValTool/tree/main/doc/sphinx/source/api

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeh lemme add all those to the list that way then

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it works nicely! Less than two minutes to run, but it throws 100+ warnings, see https://readthedocs.org/projects/esmvaltool/builds/24660091/ - I think I know what's going on in here - there are two types of warnings:

  • a lot coming from autodoc which I should turn it off completely
  • another lot coming from autoapi docs dir sitting inside our source so it's kicking its own butt since the autoapi docs are not included in any docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey Bouwe, any ideas how to fix these warnings? https://readthedocs.org/projects/esmvaltool/builds/24661254/ - I've done everything I could think of, and a tap dance on top of it too - autodoc keeps wanting to import modules, even though I added ESMValTool and esmvaltool to the Python path (that's just not how things should be done, but that's what the Sphinx folks say we should do), and the blithering autoapi keeps creating that index.rst even if I told it no with autoapi_generate_api_docs = False (see https://sphinx-autoapi.readthedocs.io/en/latest/how_to.html). I am stuck, and am gonna leave it for now 😁

Copy link
Member

@bouweandela bouweandela Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I fixed the build in f07c760, but it is not pretty. Somehow I cannot get autoapi to respect what is in __all__ (see here). At least in part that seems to be because it also searches for executable scripts and documents those and those appear to be scattered throughout the code base. Maybe if we clean that up things will work better?

We would still need to move the documentation from the now deleted doc/sphinx/source/api directory to the Python modules and do some further polishing before this is ready.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a serious bit of progress - from 126 to 7 warnings, bud 🍺 But what do you mean moving the docs to the Python modules, as in create the docs in the same place where the source code lives? Lemme see about restricting Autoapi to not look for executables, that should have a configuration solution

Copy link
Member

@bouweandela bouweandela Jun 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not executable scripts after all, it is missing __init__.py files that generate all the extra packages. Let me try to add some.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahaa! That's me completely unawares - it's great you figured that out otherwise it'd have taken me until 2027 to 😁 So I figured out the tests issue and now I think I just need to add Katja's testkw to the ignored list too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But what do you mean moving the docs to the Python modules, as in create the docs in the same place where the source code lives?

Copy the text from the deleted .rst files to the module docstrings

autoapi_type = 'python'
autoapi_file_pattern = "*.py"
autoapi_options = ['members', 'private-members', 'show-inheritance',
'show-module-summary', 'special-members', 'imported-members', ]
exclude_patterns = ['diag_scripts']

# Autodoc configuration (obsolete)
autodoc_default_options = {
'members': True,
'undoc-members': True,
Expand Down Expand Up @@ -119,7 +129,7 @@

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# exclude_patterns = []

# The reST default role (used for this markup: `text`) to use for all
# documents.
Expand Down
2 changes: 2 additions & 0 deletions doc/sphinx/source/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
pydata-sphinx-theme
sphinx-autoapi