emmylua annotations -> luals luaCATS conversion

[jamestrew]

Is a desire to convert the emmylua based type annotation grammar to luals' luacats grammar?
Considering the growth in popularity of luals' luacats, I think this would be nice.

If there is, I can take a stab at it. I have some experience writing tree-sitter grammar.

[mikavilpas]

I'd like to offer help if there's any interest in that. I have some experience with parsers, mostly parser generators though. I also maintain https://github.com/GregRos/parjs which is a parser combinator library for javascript / typescript.

Some projects publish two grammars, such as https://github.com/tree-sitter-grammars/tree-sitter-markdown

Maybe this could be a good option for this project as well. I'm thinking a new tree-sitter grammar could be created for https://luals.github.io/wiki/annotations/

Any thoughts?

[jamestrew]

I actually started this a few months back but it's been on hold for a while. Trouble is, the new LuaCATS syntax can be pretty whitespace sensitive and it's a hassle integrating it into the existing parser. And I'm unsure if just writing an annotation-only (and not parse lua code) tree-sitter parser or even not using tree-sitter entirely would be better.

I've also been following neovim core's rework of their docgen and integration of LuaCATS (and talked a little with Lewis on the core team who's been working on this). When I initially created this issue, there was still some ideas of using tree-sitter but since then, Lewis has written a LuaCATS parser using lua and LPEG but told me there aren't plans to make this accessible to the wider neovim ecosystem. At least at the time of the convo several months before more recent changes. Not sure if these plans have changed since.

But at this point, considering the work Lewis has done, it might be easier to fork the work done by Lewis. Or at least revisit this discussion with him.

[mikavilpas]

It seems like what we want should be done in two steps:

1. parse LuaCATS source code (documentation) into a machine readable format
2. generate documentation based on what was parsed

I guess the hard part is 1 (parsing), so for that, I'd like to offer two alternative options to writing our own parser.

Idea 1

Use the grammar from https://github.com/tree-sitter-grammars/tree-sitter-luadoc. I believe the grammar is accessible from https://www.npmjs.com/package/tree-sitter-luadoc as an npm package. I think this is what my neovim uses to highlight luacats annotations already, and in my experience it has worked well for that.

Idea 2

Use the lua_ls project's "export documentation" feature to get access to documentation as JSON https://luals.github.io/wiki/export-docs/

---

Here's some thoughts about each option:

Idea 1

• 👍🏻 the developer experience of migrating to LuaCATS would be nice: convert documentation -> verify that syntax highlighting still works -> verify that the LSP can provide documentation correctly. I just did this yesterday in my PR, linked above 🙂
• 👍🏻 provides additional testing which will benefit the tree-sitter-luadoc project, and also the neovim community by extension
• 👍🏻 compared to idea 2, gives us more control over the parsing, since we can define more of the documentation generation logic
• 👎🏻 on the other hand, there's more to do
• 👍🏻 since this project already uses tree-sitter there's a possibility to migrate entirely to using this grammar (can't say if this is desirable though)

Idea 2

• 👍🏻 less to do
• 👍🏻 very easy to stay up to date with the features of the lua LSP since it's literally the same project
• 👎🏻 I can't see a clean way of adding a dependency to the lua_ls project besides git submodules, which are more of a hassle to manage than npm packages
• 👎🏻 we can't customize the documentation generation logic that much (might also not be an issue)
• 👍🏻👎🏻 the implementation is completely separate from the existing implementation in this project

[Conni2461]

Idea 1 sounds better, easier to migrate. I wanna keep most of the docgen code here, it has some small issues (most of them are related to the current parser) so it should be straight forward to migrate. The only question is if we want to do this here or in a separate repository, because this was always an alternative lua parser and now switching the parser to different one is also kinda weird.

I'd like some input from @tjdevries

[tjdevries]

I don't know that I have a strong opinion on it right now, I would have to read quite a bit more of the links in the thread

[jamestrew]

I considered tree-sitter-luadoc as well but I have a few reservations about using it directly

• I don't think we can use just it on its own - which mean also including the other tree-sitter-lua parser
• adding custom annotations specifically for vimdoc "metadata" like @brief, @tag means even more parsing on top
• no super sure about the extent of LuaCATS support - from what I can tell it's a hybrid of emmylua and luacats

But at this point, considering the work Lewis has done, it might be easier to fork the work done by Lewis. Or at least revisit this discussion with him.

Spoke with Lewis about this again and he's in support of someone productizing a lot of the work his done for neovim core's docgen into something more fit general use. I think I'll give this approach a try unless there's some objection.

[CppCXY]

Would you consider using the library I wrote? It is written in Rust, and I ensure its syntax correctness

see: https://crates.io/crates/emmylua_parser

[jamestrew]

So I actually implemented a docgen script/plugin effectively forking neovim core's docgen scripts.
https://github.com/jamestrew/docgen.nvim

It parses LuaCATS (+ a few extra custom annotations) strictly using lua (LPEG) and uses the markdown treesitter parser that comes shipped with neovim so generate docs. So there's no external dependencies or build steps.
I've been using it in telescope-file-browser for a file.
I made a PR in telescope to switch it from tree-sitter-lua nvim-telescope/telescope.nvim#3227