automated build of protobuf documentation missing

[ArcEye]

Issue by mhaberler
Thu Jun 16 06:41:49 2016
Originally opened as machinekit/machinekit#970

---

this is an anchor issue for integrating the machinekit protobuf docs into the website

it's a sub-project of machinekit/machinekit#924 which is getting a tad long, so let's focus related work here

my overall idea is:

• the protobuf definitions are all important and need to be documented - ideally the same way like all the rest, 'Edit this page' link inlcuded on a per-proto-file basis
• there is a neat tool - protoc-gen-docs - which aids in generating docs from proto files - see input example and resulting output
• protoc-gen-docs is already part of the jekyll website build container and running on jenkins.machinekit.io

So to integrate the protobuf docs build into the website I guess we need to:

• add the protobuf repo into the machinekit-docs repo as a subtree (just as with mk/mk - no magic, explained here)
• add a Makefile step which runs protoc-gen-docs on the proto files, generating asciidoc; that is in place already and just needs to be integrated, and proper place for the output found
• adapt the Mustache template to output proper asciidoc - this file needs translating from markdown to asciidoc syntax ; the current output formats do not include asciidoc yet
• think through how 'Edit this proto file' links can be massaged into the output
• link the protobuf docs page in a warm place (likely near manual pages)

The build process then would be like so:

• update the protobuf git subtree
• run the protobuf-docs make target which does the protoc-gen-docs job
• run the website build

I will take care of the jenkins work to achieve that, but a sharing work with the template work would be great

[ArcEye]

Comment by ArcEye
Thu Jun 16 08:57:52 2016

---

I have updated my pull to include protobuf.

Will see what running make generates and go from there.

First thought is that the whole Table of Contents section at the top of the template is unnecessary.
So long as the file contains sections and subsections, just having :toc: beneath the first section header will produce it automatically

[ArcEye]

Comment by ArcEye
Thu Jun 16 10:47:15 2016

---

Pretty much done

Assuming the :ignore-front-matter: is still being added by the docs build, asciidoc.mustache now renders a asciidoc like so
https://github.com/ArcEye/machinekit/blob/master/src/machinetalk/proto/build/doc/machinetalk-protobuf.asciidoc
(if you view it as 'Raw' the TOC will be rendered - if you have the asciidoc plugin in your browser)

Which converts into a html with TOC like so
https://github.com/ArcEye/machinekit/blob/master/src/machinetalk/proto/build/doc/machinetalk-protobuf.html
(you will need view it here
http://mgware.co.uk/temp/machinetalk-protobuf.html
why on earth github refuses to just display html docs properly, I don't know)

[ArcEye]

Comment by ArcEye
Thu Jun 16 13:55:11 2016

---

Finally found a way to make all types into links, so that clicking the type within a table, takes you to either the entry in the table of standard data types or the special type definition header.

One way which worked perfectly in asciidoc, ceased working once the document was converted to html 😝

Links above updated to view results

[ArcEye]

Comment by ArcEye
Thu Jun 16 14:04:49 2016

---

think through how 'Edit this proto file' links can be massaged into the output

Edit this page is meaningless for this document.
Since it is completely generated, you have to either edit the template or the actual source code.

Suggest a rider on the document to that effect, unless we can think of a way to disable the Edit this page link when this document is displayed.

[ArcEye]

Comment by mhaberler
Thu Jun 16 15:03:31 2016

---

hey, superb progress!

"Edit this page" might refer to the template - does that make sense?

otherwise it'd be great to have an edit link per-proto-file, that would make it super-easy to touchup on them/document them - the proto edits would go to mk/machinekit-protobuf

[ArcEye]

Comment by ArcEye
Thu Jun 16 15:34:52 2016

---

"Edit this page" might refer to the template - does that make sense?

Not really, since the template has no control over the content, just the format and order it is displayed in.
It is available for PR like any other file

otherwise it'd be great to have an edit link per-proto-file, that would make it super-easy to touchup on them/document them - the proto edits would go to mk/machinekit-protobuf

If you can think of a way to do that's great.
{{page.edit-repo}} and {{page.path}} are not going to point there.

The situation is just the same for the generated man pages, editing the page is pointless as the component is in another repo and needs to be subject of a proper PR, not just a document presentation tweak.

I think we just have to be realistic, if someone actually knows something about protobuf or components, they should be able to make a PR in the normal way.

[ArcEye]

Comment by ArcEye
Thu Jun 16 15:40:10 2016

---

The main thing that needs doing in the .proto files is adding descriptions in a way that will get picked up by proto-gen-doc.

That needs doing by someone who actually knows about them.

You will note that every Description field in the tables is currently empty.

Use /** */ or /// comments to document your files. 
Comments for files go at the very top of the the file. 
Comments for enumerations, messages and services go before the message, enumeration or service definition. 
Comments for fields, enum values, extensions and service methods can go either before or after the definition. 
If a documentation comment begins with @exclude, the corresponding item will be excluded from the generated documentation.

[ArcEye]

Comment by mhaberler
Fri Jun 17 06:49:30 2016

---

congratulatios - pseudomuto/protoc-gen-doc#171 !

[ArcEye]

Comment by mhaberler
Fri Jun 17 06:54:11 2016

---

I defintely will take on cleaning up the protos and adding proper comments

being a lazy bastard I just thought we can work in an additional per-proto link to like for instance https://github.com/machinekit/machinetalk-protobuf/blob/master/src/machinetalk/protobuf/rtapicommand.proto

anyway let's get this into the build process first and rearrange deckchairs second - what do I need to do?

[ArcEye]

Comment by ArcEye
Fri Jun 17 07:39:45 2016

---

congratulatios - pseudomuto/protoc-gen-doc#171 !

It is a good system and surprisingly easy to get professional results
Us moustachios have to stick together 😄

[ArcEye]

Comment by ArcEye
Fri Jun 17 07:50:44 2016

---

anyway let's get this into the build process first and rearrange deckchairs second -

I will do a PR for the template shortly, but it will be subject to change to get the descriptions in

what do I need to do?

I would suggest just take one file for now and annotate as per the quoted text, 3 posts above.

Try a general description in a /** */ block at the top of the file
then comments behind /// before all functions, enumerations, messages, services, fields, enum values, extensions and service methods.

That will give a known format to work with and we can see what proto-gen-doc makes of it and how best to set the template to display it.

[ArcEye]

Comment by ArcEye
Fri Jun 17 08:22:17 2016

---

Doing some experiments.

message Emc_Traj_Linear_Move {

    option (nanopb_msgopt).msgid = 103;
    /// description of something
    required MotionType   type = 10;
    required EmcPose      end = 20;
    required double        vel = 30;
    required double ini_maxvel = 40;
    required double        acc = 50;
    required bool    feed_mode = 60;
    required int32 indexrotary = 70;
}

will produce description of something in the description field for MotionType

So contrary to the docs, the description must appear immediately above (before) the field it is describing.

Putting the same comment below indexrotary produces nothing

[ArcEye]

Comment by ArcEye
Fri Jun 17 08:56:15 2016

---

More experimentation

File descriptions MUST start at line 1, col 1

They can either be block commented with /** */ or a single /// if just one line
If you want a new line in the text, must double space or they will run into each other

I will PR the amended template

Following this format produces:

[ArcEye]

Comment by mhaberler
Fri Jun 17 10:04:20 2016

---

super!

thinking how to do this in jenkins

as a first step I added machinetalk-protobuf as a subtree to the docs dir: https://github.com/mhaberler/machinekit-docs/tree/protobuf-subtree

commands how it was created + how to update are in the commit message

@ArcEye - if this is to your liking, I'll make it a PR

the Makefile would do the git subtree pull ...
then create the protobuf asciidoc file as a temp file, compare it to the one in the tree, and update it if they dont (to save unnecessary rebuilds)

sounds about right?

[ArcEye]

Comment by mhaberler
Fri Jun 17 14:55:43 2016

---

I must have botched that branch above

this looks more like it: machinekit/machinekit-docs#166

frontmatter still missing

[ArcEye]

Comment by mhaberler
Sat Jun 25 14:49:54 2016

---

it seems to me we're next to done with the plumbing part - the editing of the protobuf messages itself of course not withstanding?

[ArcEye]

Comment by ArcEye
Sat Jun 25 15:01:28 2016

---

Looks like it, they seem to build fine

[ArcEye]

Comment by ArcEye
Wed Jul 13 08:24:01 2016

---

@Strahlex
Don't know if you saw this, as you were away at the time

Whilst you are working upon the protobuf files, be aware of the documenting conventions as above at machinekit/machinekit#970 (comment)

If you use them, anything thus marked will be automatically rendered in the docs