Odoc 3.0 Planning: Structured Docs, References, Breadcrumbs and Sidebar

[jonludlam]

Hi everyone! We (the Odoc team here at Tarides) been thinking a lot recently about Odoc and the direction it's going. There have been a number of new features recently and some in the pipeline that made us want to write up our thoughts, and check with the community about the way we're heading.

The following is a write-up of our internal thoughts on the subject, where we've got to the point where we've got something more-or-less coherent that it will be useful to share with a wider audience to seek feedback on the ideas.

This isn't an exhaustive list of everything we'd like to do! However, this represents in some sense the 'core' of the design that it's important and most tricky to get right. We'll also post some more notes later for other aspects of the design (e.g. source rendering and others).

Introduction

This document describes a number of features geared towards improving the experience of documenting OCaml projects.

Included are a number of feature suggestions that are entirely new to odoc, some that have only recently been added, and some adjustments to existing features as a result of the experience of working with them.

First, we define an extension to the existing description of a package's documentation in the odig docs. The concept described there is of a documentation set - a 'unit of documentation', consisting of a number of documentation files and associated assets in addition to metadata about them. There may also be a number of associated OCaml libraries for which API documentation from doc comments are considered to be part of the documentation unit.

The extensions described here are:

1. to allow structured docs - ie, subdirectories and metadata describing the sidebar,
2. to explicitly describe the scope of allowable references,
3. to add the facility of referencing docs and libraries of other packages,
4. to include rendered source code as an integral part of the documentation set (detail of this to follow in a separate doc).

As part of this, we will also describe the construction of a sidebar for navigating the documentation set.

We also take this opportunity when implementing these to address the way that odoc is driven to allow for better incremental building, including subsets of available docs. The details of these changes will follow in a further doc.

Principles

1. We aim to not break existing documentation.
2. It must be possible for the documentation to build as expected both for hosting on ocaml.org and by "local drivers"
3. Documentation should be buildable from "installed sources", and not require additional metadata not installed.

Globally-relevant information

This section contains information that is relevant to everyone maintaining ocaml package. These are the changes that will end up affecting the contents of peoples repositories, baked into package releases, and which we will do our best to support in odoc for the forseeable future.

Structured documentation

A 'unit of documentation' consists of a set of documents and assets in a hierarchical directory structure. Documentation files are mld files, and assets are jpegs, pngs, svgs, etc., that are referenced by the docs.

For opam packages, there is one unit of documentation associated with each package, and the mlds are installed when installing a package with opam into <opam dir>/doc/<package>/odoc-pages/ and subdirectories.

Each documentation directory or subdirectory should contain an index.mld file that represents the main index of the pages and subdirectories found within.

In addition to the the text of the page, the index.mld files should contain a list of the pages and subdirectories that will be used to order the pages in a sidebar. If this does not exist, the order will be done by an appropriate sort. The order can be specified in the "frontmatter" or metadata block associated with the index.mld file (defined later).

Assets may be put alongside the mld files, or they may be in a separate directory <opam dir>/doc/<package>/odoc-assets/. There will be new syntax for referencing assets via new elements of odoc syntax such as images.

References and scoping

It is implicit that all of the libraries of the package are 'in scope' when compiling/linking the documentation for a package. That is, all of the modules of all of the libraries are accessible via the usual syntax of {!Module}. Extra libraries (including, but not limited to dependencies of the packages libraries) will only be able to be referenced by explicitly requesting it. Only those libraries of the package that can be or have been successfully compiled will be linkable.

Pages in other packages may be referenced, but once again they need to be explicitly requested. Requesting a package's pages and requesting libraries are independent.

Concretely, the extra packages and libraries will be added to fields in the metadata block (defined later).

We don't want to restrict documentation authors to the same constraints that OCaml programs and libraries must stick to, so where there are libraries that cannot be linked together in the same program we would like to still allow that when linking docs. A consequence of that is that modules need to be available in a namespaced way as well as via the standard flat namespace.

• A package can be added to the scope, which will make that packages pages accessible by the reference {!/packagename/a/b/c/page-d} or more simply {!/packagename/a/b/c/d} which would refer to the page that usually ends up installed at <opam dir>/doc/<packagename>/odoc-pages/a/b/c/d.mld

• A module can be added to the scope by adding the library it's found in, which will make those modules accessible by the references {!Module} or {!/libname.sublib/Module} (not {!sublib/Module}). By default all modules in the package's libraries are added to the scope.

Note

It's not always easy for anyone to determine in which library a particular module lives, which is a problem that current drivers have to worry about, and which this document does not change. If there's only one library that adds a specific include path to the compiler invocation then it's easy. If not, we have to fall back to querying archives for modules, although this only works for modules that have an implementation.

References in pages

When resolving references in a page within the hierarchy, pages can be resolved by different mechanism:

1. By name where all pages of the current package are available, for references that do not contain a /: {!pagename}, {!page-pagename}.
2. By relative or absolute references, up to the root of the documentation tree, for references that contain a /: {!dirname/pagename}, {!../dirname/pagename}, {!packagename/dirname/pagename}.

If two pages have the same name, they cannot be referenced by name.
To resolve a page in your own directory using a "relative path" reference, one need to use {!./pagename} instead of {!pagename}: otherwise, there is no / and the "resolving by name" is used.
Finally, note that there is no way to use a resolving by name for pages in another package.

For example, given the hierarchy (omitting index.mld files):

/foo.mld
/a/bar.mld
/a/b/baz.mld
/a/b/qux.mld
/a/b/d/quux.mld
/a/c/fun.mld

when resolving references in baz.mld, the following are valid references (assuming our package name is pkgname):

{!../page-bar} or {!../bar}
{!../c/page-fun} or {!../c/fun}
{!../../page-foo}
{!d/quux}
{!dir-d/quux}
{!qux}
{!./qux}
{!/pkgname/page-foo} or {!/pkgname/foo}
{!/otherpackage/page-something_in_another_package}

At a lexical level, this will be handled by considering a reference to be optionally prefixed by a path terminating in a /, except where that / is in parentheses. The conditional is to handle references to infix values containing /, e.g. {!val-(/)}.

References in modules

• A package can be added to the scope, which will make that packages pages accessible by the reference {!/packagename/a/b/c/page-d} or more simply {!/packagename/a/b/c/d} which would refer to the page that usually ends up installed at <opam dir>/doc/<packagename>/odoc-pages/a/b/c/d.mld

• If there are documentation pages for the package in which this module is contained, these are also added to the scope: they are linkable with the reference {!pagename}. If two pages have the same name, they need the {!/pkgname/path/page} syntax to be referenced.

• Modules that are in scope are exactly those in libraries that are dependencies of the module in question. No more, no less. These are accessible by the references {!Module} or {!/libname.sublib/Module} (not {!sublib.Module}).

Metadata block

The new items of metadata required by the above will be put into a metadata block somewhere near the beginning of the file. The suggested format for this is a code block with the language meta, containing key-value pairs:

{@meta[
packages: ocamlfind re
libs: findlib astring re re.emacs
toc: page1 page2 dir-examples page3
]}

This is similar to the frontmatter of markdown docs, although we (currently) don't propose to have full yaml in the blocks, just key-value pairs. Only three keys are defined currently; packages, libs and toc, but extra keys will be allowed for extensibility.

Note that it's out of scope for this document how the packages get installed. We expect there to be constraints on what is possible, and there should be discussion on these at a later point.

Driver-level changes

The following sections won't affect what's in ocaml packages, but is important for consumers of the documentation as well as people maintaining tools that drive odoc in various ways.

Output file structure

In odoc today, the modules and pages are output into the same hierarchy when using the --package argument, although the parent/child arguments allows the output to be put wherever you like.

In Odoc 3, we will be adding another mechanism that allows for the specification of the output path, but without some of the drawbacks of the parent/child mechanism.

In all current drivers; odig, dune and voodoo (for ocaml.org), the mld pages exist in the same hierarchy as the module reference documentation. With the addition of structure to the mld pages, and the addition of another hierarchy of source pages, there now exists a much more likely danger of filename clashes. We therefore propose to allow that the three hierarchies no longer necessarily have to be output under the same path, so for example we would like it to be possible for the resulting pages to exist under the URLs /lib for the module reference documentation, /doc for the mld-derived pages, and /src for the source hierarchy.

While it must be possible for the hierarchies to be different, it also must be possible to have them share the same path. For example, it may make sense to keep the manuals and the modules docs under the same tree.

Breadcrumbs

The breadcrumbs at the top of the page are currently purely based on the directory structure of the output files. We propose to change this such that in two few ways.

1. We still have a one-to-one mapping of URL directory to item in the breadcrumbs, but only up to a specified top-level directory
2. The actual text in the breadcrumbs are taken from the documents present at the URL pointed to

Manual pages

For example, where we have a page on the ocaml.org site like:

/p/package/1.0/doc/tutorial/simple.html

we might have the breadcrumbs

Package 1.0 Info > Manuals > Tutorial > Simple

where the name Simple comes from the {0 Simple} inside odoc-pages/tutorial/simple.mld, Tutorial comes from the {0 Tutorial} in odoc-pages/tutorial/index.mld, Manuals comes from the {0 Manuals} in odoc-pages/index.mld, and Package 1.0 Info would come from the file generated by the driver representing the package metadata page.

Note that this does not require that the top page be one generated by a driver, it could equally be one whose text comes from odoc-pages/index.mld. However, in ocaml.org today the index.mld page is found at the doc subdirectory of the main package page, and we would like to support this case.

Since the title might be lengthy, we will also add some syntax to allow for shorter names:

{{0 Short Title}A longer and more descriptive title}

Modules

Another example will be a module found in a library. On ocaml.org, these currently live in the root of the documentation tree, so for example, there is a module page that lives at this URL:

/p/re/1.10.4/doc/Re_emacs/index.html

When visiting that page today, we see the breadcrumbs:

Documentation > re.emacs (lib) > Re_emacs (Module)

so the breadcrumbs do not reflect the directory structure. In this proposal, the module would be output into the path:

/p/re/1.10.4/lib/re.emacs/Re_emacs/index.html

and the breadcrumbs would reflect that:

Re 1.10.4 info > Reference > re.emacs (lib) > Re_emacs (Module)

Sidebar

The sidebar will be generated by creating an index of all pages (including modules and source) underneath a particular output hierarchy. It may include the contents of those pages (i.e., the sections in an mld file or the modules, values, type etc. found in a module). There will be options available to choose how this will be rendered, including whether siblings of ancestors can be expanded or not and more. This index will be created after linking (so a complete picture of the contents of modules can be found), and passed to the html-generate command. The details of this will be described in a further document.

[sabine]

The new items of metadata required by the above will be put into a metadata block somewhere near the beginning of the file.

Does this mean we require package authors to list all the libraries the package offers? And if they don't provide this metadata, odoc makes a best effort in discovering these?

we would like it to be possible for the resulting pages to exist under the URLs /lib for the module reference documentation, /doc for the mld-derived pages, and /src for the source hierarchy.

This makes it impossible to establish redirects for the old module reference URLs on ocaml.org. Can we use something other than /doc? You could use /docs instead. For the module reference documentation, I would recommend /api because this is in line with the URLs of the Standard Library API reference. So you would have /src, /docs, /api. However, if you keep both manual pages and module reference in the same folder, sticking to /docs and /src (for the source views) looks fine to me.

/p/re/1.10.4/lib/re.emacs/Re_emacs/index.html

If I am not mistaken, this will break the old documentation URLs on ocaml.org in a way that we cannot create redirects for.

However, I think that should not stop you from making these changes. We are still early in terms of OCaml adoption, and including the concept of libraries within a package in odoc does look like a good idea to me. We may lose a few backlinks into specific pages of package documentation, but I expect that this cost is worth to take here.

We are currently suffering significant complexity because of the unclear relationship between package and libraries and modules contained therein. Introducing these concepts at the level of odoc and placing discovery of libraries into the responsibility of the driver will simplify things downstream at voodoo+ocaml.org.

[jonludlam]

Does this mean we require package authors to list all the libraries the package offers? And if they don't provide this metadata, odoc makes a best effort in discovering these?

Not for the libraries in the package being documented, these are added by default (in order to not break existing docs!)

This makes it impossible to establish redirects for the old module reference URLs on ocaml.org. Can we use something other than /doc?

I had it in mind that we could do redirects, but that would require knowing the mapping from old url to new url, and it would not necessarily work if there were clashes. If we did a different URL for manuals, like /docs as you suggest, we could have the old url redirect to a search perhaps, so we'd remove the need to know the mapping? I'm not sure what's best.

For the module reference documentation, I would recommend /api

We chose lib as the next thing in the URL is actually the name of the library. I was hoping this would go some way to help with the package name / library name confusion that you mention!

If I am not mistaken, this will break the old documentation URLs on ocaml.org in a way that we cannot create redirects for.

Again I think we probably could at the expense of a large static map. If that's too costly we could do the search (or some heuristic in between!)

[sabine]

Does this mean we require package authors to list all the libraries the package offers? And if they don't provide this metadata, odoc makes a best effort in discovering these?

Not for the libraries in the package being documented, these are added by default

I'm curious who is adding/discovering them? The driver? Either way, as long as downstream (voodoo) does not need to do library discovery anymore, this is a nice win!

[jonludlam]

I'm curious who is adding/discovering them? The driver?

The driver will be responsible for adding them, it's not something odoc-the-tool is best placed figure out on its own.

[gadmm]

Respective to my experience and my current needs:

• Supporting assets (e.g. images) is the one important missing feature. (Currently, I use the alt of the missing image to redirect to my own version of the documentation: https://v3.ocaml.org/p/memprof-limits/0.2.1/doc/statistical.html)
• Despite a 3.0 name (major semver increment), the goal seems to not break current documentation. I don't mind about symbolic numbers but backwards-compatibility is indeed important.

[panglesd]

I think the major semver increment is due to the fact the API will be broken: odoc drivers will have to be updated!

[jonludlam]

And indeed, assets will be supported. We do very much care about backward compatibility - the most recent version of odoc is being used to create the docs for ocaml.org, so it's being run on all versions of all packages. One consequence of this is that we're trying to create docs for packages released many years ago. A second consequence is that we have to remain compatible with quite old versions of OCaml - 4.02 is our lower bound currently - and this constraint does tend to keep reminding us about compatibility :-)

Also, I'm not completely convinced that we'll have to break the API, but it certainly is the case that in order to support the new features outlined above the drivers will need updating.

[sabine]

Is there any chance, odoc 3.0 can support Markdown, by the way?

If a package only has Markdown documentation, we won't render it with odoc, right? In the long run it's obviously nice if people move to .mld because it's a more powerful format, but realistically, there's a good deal of packages that have Markdown docs.

[jonludlam]

That's something we've been thinking about too. There are a variety of things we can do, and it's likely we'll do something on the that spectrum, but we'll need to think carefully of the costs and consequences.

[jonludlam]

@dbuenzli has written some really interesting thoughts on the subject here

[sabine]

I see, that's a tough spot to be in with markdown and ocamldoc. It looks to me like that needs to be a discussion on its own.

[panglesd]

About the resolution of pages references, @dbuenzli has made some comments in the asset discussion.

[dbuenzli]

So what's the syntax for reliably accessing the landing index.mld page of my package without mentioning the package name ?

[panglesd]

There is no such syntax in the current proposal...

[dbuenzli]

Why ? Is it based on a technical difficulty ? I think the case I made for here should be compelling enough. I even proposed a syntax (to which I'm not attached but fits quite well in the picture in my opinion).

I think that package absolute links are a must.

Anyone who ever had to migrate to another domain the CMS of a website whose domain name has been hard-coded all over the place will understand :–)

[panglesd]

I personally agree. I think @jonludlam was keen on not having it in the current proposal, but adding it later if there is demand. But I might not remember his case faithfully!

(With my alternative proposal, this is just {!/index} 🙈 )

[dbuenzli]

but adding it later if there is demand.

That's the kind of thing were people will gleefully workaround by hardcoding roots than taking the time to demand. And by the time the feature will be in it will be too late…

(With my alternative proposal, this is just {!/index} 🙈 )

I wouldn't mind having your proposal. I think that the ability to have logical links in other packages is quite a good idea from a distributed documentation maintenance point of view. Especially since I expect that except for these index.mld most packages will have rather uniquely identified assets.

[dbuenzli]

Here are a few comments on quick a read of the proposal:

1. I don't think I entirely got where this metadata block is supposed to be and what exactly has to be mentioned. But it's likely that if I'm making references on other packages I don't want to repeat that information in every mld document. Shouldn't this simply contain toc for .mld pages and the other field rather live in a global metadata file, e.g. at the root of odoc-pages, or even, in META files (which are extensible) ?
2. As a driver author I don't want to parse this metadata block. If the driver will need that info odoc should be able to extract it for me in a simple from like e.g. we have for deps (I don't mind lines based, json or sexp).
3. I'm wondering if, rather than installing .mld we should not start installing some form of compiled, say .mldt, files, this could help for example with issue Warnings to unresolved references make odoc unusable #1120
4. I'm not sure I fully understood the discussion about libraries and bread crumbs. But just bear in mind that most of the time having people go through index page -> library page -> module is going to be horribly clicky and unsusable¹. In 95% of the cases it will be more than enough to have one section per library on the package landing page with and their module indexes here. Here are a few examples of what I consider reasonably usable landing pages brr, webs, zipc, bytesrw.

Footnotes

1. If you ever tried to use dune automatically generated docsets you should understand what I mean :-) ↩