Update haystack-pydoc-tools to no longer use pydoc-markdown

[sjrl]

We should also consider updating/migrating our https://github.com/deepset-ai/haystack-pydoc-tools package since it relies on pydoc-markdown which is no longer being maintained (last release in 2023). The author of that package recommends looking at mkdocstrings as a possible migration path.

We recently had to pin setuptools to v81.0.0 since in version 82.0.0 they removed pkg_resources which is needed by pydoc-markdown to work.

[anakin87]

First findings

• Turned out that mkdocstrings is in maintenance mode and does not guarantee updates after 2026.
• https://github.com/daizutabi/mkapi/ is also recommended in pydoc-markdown discussions; pdoc is also interesting.
  • tested both, but primarily generate HTML and are not easily customizable

https://github.com/mkdocstrings/griffe is a library to extract Python signatures, well-maintained and popular, that could be a better fit: we could use it to parse (replace pydoc-markdown loaders) and then build a simple render ourselves (or use related projects). I'll investigate this direction.

[anakin87]

Done in the linked PRs +
#10617
deepset-ai/haystack-experimental#441
deepset-ai/haystack-core-integrations#2855