WIP: Lua documentation

[tjdevries]

WIP.

It appears that I formatted the JS file and generated a lot of changes...

@vigoux want to pick a formatter that we can run over grammar.js to make sure we have the same format?

[vigoux]

Sounds great so far ! Will it allow to parse what is inside the comments ?

[tjdevries]

Yes, when I move from C -> JS you can do something like it first hits function_comment and then you can have many sub comment types inside that match on things like @param or whatever we like.

[tjdevries]

tree-sitter-lua on  tjdevries/lua_documentation [⇡] ❯ 🐍 untitled                                     13:28:05
➜ cat ~/tmp/small.lua
local x = {}

--- hello world
--@param y: add 1
x.my_func = function(y)
  return y + 1
end

return {
  something = x
}

tree-sitter-lua on  tjdevries/lua_documentation [⇡] ❯ 🐍 untitled                                     13:28:19
➜ tree-sitter parse ~/tmp/small.lua
(program [0, 0] - [11, 0]
  (local_variable_declaration [0, 0] - [0, 12]
    (variable_declarator [0, 6] - [0, 7]
      (identifier [0, 6] - [0, 7]))
    (table [0, 10] - [0, 12]))
  (variable_declaration [2, 0] - [6, 3]
    (lua_documentation [2, 0] - [4, 0]
      (parameter_documentation [3, 0] - [4, 0]
        name: (parameter_name [3, 9] - [3, 10])
        description: (parameter_description [3, 12] - [3, 15])))
    (variable_declarator [4, 0] - [4, 9]
      (field_expression [4, 0] - [4, 9]
        (identifier [4, 0] - [4, 1])
        (property_identifier [4, 2] - [4, 9])))
    (function_definition [4, 12] - [6, 3]
      (parameters [4, 20] - [4, 23]
        (identifier [4, 21] - [4, 22]))
      (return_statement [5, 2] - [5, 14]
        (binary_operation [5, 9] - [5, 14]
          (identifier [5, 9] - [5, 10])
          (number [5, 13] - [5, 14])))))
  (module_return_statement [8, 0] - [10, 1]
    (table [8, 7] - [10, 1]
      (field [9, 2] - [9, 15]
        (identifier [9, 2] - [9, 11])
        (identifier [9, 14] - [9, 15])))))

[tjdevries]

Now requires: neovim/neovim#12388 (exposes some new non-buffer related endpoints)

Current status:

➜ VIMRUNTIME=~/build/neovim/runtime ~/build/neovim/build/bin/nvim --headless -u NORC -c "luafile ./extensions/lua_docs.lua" -c ":q"
{
  ["x.my_func"] = {
    params = { {
        desc = " add 1",
        name = "y"
      } }
  },
  ["x.other_func"] = {
    params = { {
        desc = " final param",
        name = "two_vars"
      }, {
        desc = " another value",
        name = "wow"
      } }
  }
}

[tjdevries]

from lua file:

local x = {}

--- hello world
--@param y: add 1
x.my_func = function(y)
  return y + 1
end

--- hello world
--@param wow: another value
--@param two_vars: final param
x.other_func = function(wow, two_vars)
  return wow + 2
end

return {
  something = x
}

[tjdevries]

tree-sitter-lua on  tjdevries/lua_documentation ❯ 🐍 untitled                                                                                                                     12:49:03
➜ cat ~/tmp/small.lua
local x = {}

--- hello world
--@param y: add 1
x.my_func = function(y)
  return y + 1
end

--- hello world
--@param wow: another value
--@param two_vars: final param
x['other_func'] = function(wow, two_vars)
  local private_variable = 12

  return wow + two_vars
end

local export_table = {}

--- Wow, a bar
--@param foo: thing
export_table.bar = function(foo)
end

--- Also has a description, since it's important
local other = 3

--- Also has a desc, but not important
local not_exported = true

return {
  something = x,
  other = other,

  should_not_show = private_variable
}

tree-sitter-lua on  tjdevries/lua_documentation ❯ 🐍 untitled                                                                                                                     12:49:06
➜ VIMRUNTIME=~/build/neovim/runtime ~/build/neovim/build/bin/nvim --headless -u NORC -c "luafile ./extensions/lua_docs.lua" -c ":q"
{ {
    export_table = {
      bar = {
        params = { {
            desc = " thing",
            name = "foo",
            original_parent = "export_table"
          } }
      }
    },
    something = {
      ["['other_func']"] = {
        params = { {
            desc = " another value",
            name = "wow",
            original_parent = "x"
          }, {
            desc = " final param",
            name = "two_vars",
            original_parent = "x"
          } }
      },
      my_func = {
        params = { {
            desc = " add 1",
            name = "y",
            original_parent = "x"
          } }
      }
    }
  } }%

Now sees name redirects on module returns

[vigoux]

After some thinking, I think that creating a dedicated parser for doc comments and using language injection can be better, what do you think about that ?

[tjdevries]

I'd be happy to do that if I can find a way to do it. The documentation is difficult to understand regarding the injection. All of the important parts of the parser will be easy to see and abstract whenever we want.

[vigoux]

Now that injection is on it's way, we really should be thinking about how we can separate this comment parsing in its own parser.
A difficult part is being able to ignore the language specific comment token. It should be doable though. For example, the parser could ignore everything at the start of a line that is neither a word character nor an @