Compare the coverage

We have tree ways of checking running the doctests on SciPy:

1. The old way: refguide-check
2. The doctest way: via the doctesting layer
3. The pytest-plugin way: via the pytest plugin layer.

We want to make sure that the "new" ways have at least the same coverage as the refguide-check way. So we dump the logs of all three ways, lightly postprocess to make the diffs clearer, and actually diff them.

The TL;DR is that the results are compatible. Below are gory details, to simplify rerunning similar checks if needs be.

---

The pytest plugin way

• Patch the scpdt plugin to include functions with no examples in the docstrings: comment out the if test.examples: # skip empty doctests clause at https://github.com/ev-br/scpdt/blob/main/scpdt/plugin.py#L184

• Use the scipy conftest mods from https://github.com/scipy/scipy/compare/main...ev-br:scipy:doctest_plugin (the git branch is at ev-br/scipy/doctest_plugin) to activate the plugin

• Run the plugin and stop at the collection stage:

$ python dev.py shell
$ cd path/to-scipy/folder
$ pytest build-install/lib/python3.10/site-packages/scipy/ --doctest-modules --ignore=build-install/lib/python3.10/site-packages/scipy/interpolate/_interpnd_info.py --ignore=build-install/lib/python3.10/site-packages/scipy/_lib --collect-only > plugin.log

Then run the plugin.log through a small utility to extract the names of functions and classes:

"""
convert the result of 
$ pytest ... --doctest-modules --collect-only
to simplify comparing to the doctest_.log from doctest-based runs
"""

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2:
        print("Usage: modify_log.py LOGFILE")

    fname = sys.argv[1]
    with open(fname, 'r') as inf:
        in_lines = inf.readlines()

    out_lines = []
    for line in in_lines:
        line_ = line.strip()
         
        if line_.startswith("<DoctestItem"):
            out_lines.append(line_[13:-1])

    print("\n".join(sorted(out_lines)))

---

The refguide-check way

• Use https://github.com/scipy/scipy/pull/16391 which patches the refguide-check utility so that running

$ python dev.py refguide-check

drops a file refguide_check.log in the SciPy root folder.

• Run the log through

"""
strip module names from the refguide-check.log
to simplify comparing to the doctest_.log from doctest-based runs
"""

PACKAGE = "scipy."

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2:
        print("Usage: modify_log.py LOGFILE")

    fname = sys.argv[1]
    with open(fname, 'r') as inf:
        in_lines = inf.readlines()

    out_lines = []
    for j in range(len(in_lines)-1):
        line = in_lines[j]
        next_line = in_lines[j+1]

        if line.startswith("="):
            continue

        if next_line.startswith("="):
            # this is a package name, add and reset
            out_lines.append(line)
            PACKAGE=line.strip()
            continue

        out_lines.append(f"{PACKAGE}.{line}")

    print("".join(sorted(out_lines)))

---

the doctest way

• The scipy PR: https://github.com/scipy/scipy/pull/16391 adds a dev.py doctest command:

$ python dev.py doctest

drops the doctest.log file in the SciPy root folder. Run it through

"""
strip module names from the doctest.log
to simplify comparing to the doctest_.log from doctest-based runs
"""
if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2:
        print("Usage: modify_log.py LOGFILE")

    fname = sys.argv[1]
    with open(fname, 'r') as inf:
        in_lines = inf.readlines()

    out_lines = []
    for j, line in enumerate(in_lines):
        line_ = line.strip()

        if line_.startswith("="):
            continue

        if j < len(in_lines) -1 and in_lines[j+1].startswith("="):
            continue

        out_lines.append(line_)

    print("\n".join(sorted(out_lines)))

---

The net result --- with scipy main branch at cd9d20bb30d78e1f56d8f1e9a33088c1a860446a and scpdt main at https://github.com/ev-br/scpdt/commit/9ce08b0581bf5a9e0e14e1eb1d3fc5d3271055ab --- is that the results are the same, modulo the following differences:

• refguide-check still goes through scipy.misc --- irrelevant, may be ignored
• the plugin misses scipy.fftpack.convolve --- not a show stopper, is tracked in https://github.com/ev-br/scpdt/issues/106

Here's the sorted coverage listing from the plugin: https://github.com/ev-br/scpdt/wiki/plugin-SciPy-doctest-coverage