PR: Overhaul, fix and improve docstring generation (Editor)

[CAM-Gerlach]

Description of Changes

• Wrote at least one-line docstrings (for any new functions)
• Added unit test(s) covering the changes (if testable)

• Included a screenshot or animation (if affecting the UI, see Licecap)

Currently a draft as while all code changes are complete and functionally tested, the unit tests need to be updated to match the format changes, add regression tests for the bugs fixed, and add new tests for the integration of existing docstrings and other new cases.

Builds on top of the initial formatting fixes of PR #24041 (thanks @rhkarls !), all of which are incorporated here in as first commit (with the one exception, see below). On top of that, this PR makes the following changes.

Removed limitations

• Incorporate existing docstring content as description in generated one
• Handle duplicate copy of existing docstring when generating a new one
• Parse existing docstring sections & integrate section by section into the generated docstring
• Support generating Sphinxdoc return type(s) from func body, like for the other formats

Bug fixes

• Fix generate docstring failing to work at all on Black-style function signatures
• Fix a bug where a trailing comma in the function parameter list caused an extra empty parameter to be generated
• Fix false positives with certain parameter names in the check for the start of a new function causing docstring generation to fail
• Fix multiple false positives, negatives and bugs with return and yield detection and extraction
• Fix return type annotations with certain valid characters (e.g. |, ., etc) not being recognized at all
• Fix various other edge cases causing docstring generation to not work with certain functions or work incorrectly (e.g. Docstring not generated for one particular function, Spyder 5.2.0 #16955 )
• Fix Docstrings generation is wierd for functions with many arguments #20458 , increase limit for function signature length from 20 lines to 100

Formatting fixes

• Don't expand tuples to multiple return values for Googledoc, to follow its spec
• Conversely, now do expand tuples in return annotations to multiple return values for Numpydoc, per its spec
• Format return type annotations that are multiple lines in the code as a single line in the docstring with proper PEP 8 formatting, rather than leaving loads of stray indents and whitespace
• Also reformat string annotations as normal text without the extra string quotes
• Be consistent and don't add unnecessary period & description block when generating docstrings for functions returning only None
• Fix section order in Sphinxdoc to be consistent with the others and its examples/convention (raises should go after returns)
• Remove optional from the parameter types for Sphinxdoc, where it didn't belong per the format
• Make line breaks in generated docstrings consistent between sections

Refactoring and cleanup

• Refactor docstring.py throughout to be simpler, cleaner & more Pythonic
• Refactor _generate_docstring to reduce duplication and excessive indent levels
• Refactor generate_doc() into smaller functions, one per section, and reduce duplication
• Further refactor _generate functions to use modern f-strings and build strings more cleanly, consistently and robustly
• Use single-sourced named constants to control min and max search length parameters instead of a bunch of different, inconsistent magic numbers scattered throughout the code
• Make code formatting throughout more consistent and follow Black style on touched lines (as consistency allows)
• Add and improve docstrings throughout the file
• Update file license header
• Fix unused imports and ordering

Return None behavior

The one change from PR #24041 that was not picked up here was excluding the return section entirely if the function returns None, as it leaves ambiguous whether a function actually returns None or the return type is merely not (yet) documented in its own section, as is unfortunately quite common in e.g. the Spyder codebase as well as some others, whereas including it makes that explicit. Presumably, the rationale for not requiring it in some of the docstring formats is the extra developer effort required to manually add it for every function, which is a non-issue here as it is being generated automatically.

However, I have fixed several other formatting bugs with it, specifically omitting the spurious period that was sometimes inserted after the type (None), consistently omitting the meaningless/boilerplate "DESCRIPTION" line after "None" (which previously was inserted in some cases and not others), and other minor issues. If there is sufficient interest, an opt-in preferences option to omit None return sections (for Googledoc and Numpydoc) could be added in a followup PR.

Issue(s) Resolved

Fixes #16955
Fixes #20458
Fixes #22043
Fixes #24040

Closes #24041

Will partially address some of #10622 (docstring added in multiple undo steps), not implemented yet

Affirmation

By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: CAM-Gerlach