RFC: convert to use `yard` for documentation

[flavorjones]

We're already using yard to generate the docs at https://nokogiri.org/rdoc/index.html, but we're not taking advantage of some of yard's features (like the ability to embed mixin methods, e.g., Searchable in Node and NodeSet), and we're certainly not using yard syntax (e.g., all the call-seq annotations that we currently use are ignored).

So I'm thinking it would be best to convert the doc formatting to be yard-compatible markdown, and eliminate rdoc format entirely from the codebase.

I'd love to hear comments on this proposal.

Note to future self: try using https://github.com/postmodern/hoe-yard to drive some of this; and then have the nokogiri.org docs task just call nokogiri's own rake task for the site generation.

[flavorjones]

Note that one of the big reasons I'd like to do this is to start grouping methods in large classes like XML::Node, and so I'm watching lsegal/yard#1238 which is driving the ability to use @!group in C code.

[flavorjones]

Of course, I'm noticing now that RDoc supports much of what I wanted to use yard for, including:

• :category: directive to group methods
• markdown support (not a blocker, but there it is)
• :include: directive to pull in methods from mixins (though this might already work out of the box via RDoc::Mixin? geez, I really need to read the RDoc manual)

So maybe this issue should be reviewed, and either closed or rewritten to be outcome-oriented (instead of solution-oriented).

[flavorjones]

After thinking about the foundational nature of Nokogiri in the ecosystem, I think we should use RDoc to ensure that as many folks as possible are able to generate and read the doc strings.

After reading docs, I think we can use RDoc without making it challenging for folks to navigate those docs, thanks to the :category: and :include: directives, etc.

[flavorjones]

Flip-flopping on this. I'm choosing YARD over RDoc/SDoc because:

• allows complete specification of argument types and return types
  • potentially generating Sorbet/RBI/RBS files
• looks nicer than RDoc/SDoc
• is less problematic than SDoc templates
  • SDoc template doesn't have URL magic in the browser bar like the Rails template does
  • Rails template doesn't support relative prefix so I can't deploy under https://nokogiri.org/rdoc/

[flavorjones]

Flip-flopping again, and investing in making RDoc better:

• fix: alias to a method with call-seq should render properly if the call-seq does not specify the alias ruby/rdoc#840
• feature: Render mixed-in methods and constants with --embed-mixins ruby/rdoc#842
• also thinking about noodling on a new rdoc template using something modern like Material Design or Tailwind

[flavorjones]

Note that RDoc's Markdown parser emits docs that don't look quite right throughout the RDoc toolchain (e.g., ri), so although I'd prefer to use Markdown, I'm going to use RDoc::Markup within the code.