RFC Architectural changes

[tarleb]

I've been pondering a few architectural changes to the build system and would welcome some feedback on these.

1. We are currently using flexible Dockerfiles that can be configured and adjusted to various changes. This keeps duplication at a minimum, but comes at the cost of added complexity in the Dockerfiles and the tools required to build the images. It's also sometimes difficult to recompile old images because we usually update the Dockerfile without thinking too much about re-building. However, rebuilding can be useful for security fixes and similar.

   Hence my proposal to switch to a template-based approach: the Dockerfile for each new pandoc version would be generated afresh from the template, inserting all build details.

   My initial motivation were the edge images, which require completely different settings, and currently even fail to build on alpine. Using a separate Dockerfile for edge builds would make things easier. But I think this will help in other ways, too.

   The downside is that we'd have a good bit of redundancy in the repo, with multiple Dockerfiles with minor differences.

2. Switch from shell to pandoc Lua for the build tools and helper scripts.

   While I love shell scripts and to keep everything POSIX-compatible, it seems that "dogfooding" pandoc would be a good choice, because all contributors would be at least somewhat familiar with it.

I'm curious to hear your thoughts.

[daamien]

hi @tarleb

I fully agree with the fact that the current images are hard to build and test. Lately it lead me to a few rookie mistakes ( typos, name mismatch, etc. ) that may be avoided if the build/test process was more straightforward. Also the fact that we have a limited CI minutes means we need to be careful about how we spend those minutes....

To clarify your proposal :

• Why not have a single template to generate all the different Dockerfiles and avoid redundancy ?
• then the repo would not contain any dockerfiles or the generated dockerfiles would be committed to the repo ?

I have no knowledge of Lua but I hate shell scripts from the bottom of my heart so I'm willing to give a try 😛

Also can we share examples of similar projects and how they handle this ? I was thinking especially of the postgres images
project which provides 20+ different variants....

https://github.com/docker-library/postgres

[tarleb]

Thank you for the feedback! Yes, the postgres repo is very close to what I have in mind.

I believe it should be possible to use the Docker Build Cloud only for releases, and to rely on GitHub machines for basic CI tests. That would make things a lot easier.

I will give it go then, unless @alerque objects.

[tarleb]

My development version is in branch dockerfiles-from-templates. I'm quite happy with the current progress, and have successfully rebuilt a number of outdated images using the new setup. The most interesting change, beside the generated Dockerfiles, is the file releases.yaml, which takes over the job of versions.md, but adds more info about the versions in our addon images.

There still remains a good bit of work before this can be merged, but I hope to finish the work later this week.

[tarleb]

My current approach is to generate all Dockerfiles for each version, and to make the Dockerfile independent from any external files. Additionally to the Dockerfiles, we now also place a docker-bake.json file in the version directory, and use that to define all tags, labels, etc.

The goal is to make the final build process extremely simple. Almost all of the complexity will be handled during the dockerfile generation step.