Create a sitemap after build

[tomschr]

Situation

We currently miss a sitemap.xml.

"Sitemap is a protocol in XML format meant for a webmaster to inform search engines about URLs on a website that are available for web crawling." — Source

Use Case

When a build has been finished, the script generates a sitemap.xml file. This is needed for search engines.

Possible Implementation

When a built has been finished, the sitemap is created. In the build subcommand, we could have a flag --sitemap/--no-sitemap which builds or not build the sitemap.

Things to clarify/discuss

Some open questions to discuss:

• Do we want to automatically build the sitemap by default? That mean, the --sitemap flag would be true.
• Do we want to build it for a subset of the docs? If a user requests to build smart/systems-management should the sitemap still be generated?
• Do we want to configure/set a default via the user's config?
• Do we want to configure the exclusion of files and directories via the user's config file? Or should it be in the env config file? Or should it be configurable at all?

Details

We probably want a separate command docbuild sitemap that deals exclusively with building the sitemap. This makes it possible to trigger a rebuild of the sitemap without building the whole documentation.

To build the sitemap, the script needs to:

• traverse the target tree
• exclude languages other than English
• exclude unwanted files
• rewrite the found files into an URL
• rewrite the URL into another URL (exclude the external-tree part?)
• has to do all these task in an async/threaded way.

See also

The idea was already integrated in GL://susedoc/docserv-convenience-binaries/build-sitemap.py