Support reproducible builds for Debian

[AkihiroSuda]

See:

• Reproducible builds official-images#16044

---

• https://github.com/reproducible-containers/repro-sources-list.sh
  is added to allow reconfiguring /etc/apt to use
  snapshot[-cloudflare].debian.org/archive/debian/<SOURCE_DATE_EPOCH>.
  However, this is disabled by default due to the slowness of the
  snapshot server.
  Reproducers may opt-in to enable this by specifying
  --secret id=enable-repro-sources-list,source=/dev/null.
  This is treated as a "secret" to hide it from build history for the
  reproducibility of the OCI config blob, but its content is not confidential.

• /var/log/* is removed as they contain timestamps

• /var/cache/ldconfig/aux-cache is removed as they contain inode
  number, etc.

• wget is executed with --no-hsts to disable creating /root/.wget-hsts

• To reproduce file timestamps, reproducers have to specify rewrite-timestamp=true
  as an exporter option (--output). This needs the master branch of
  BuildKit (will be included in BuildKit v0.13).

• The base image is not pinned by the hash.
  Reproducers may use the CONVERT action of source policies.
  https://github.com/moby/buildkit/blob/v0.13.0-beta1/docs/build-repro.md

[AkihiroSuda]

Result of the current revision of the PR (https://github.com/AkihiroSuda/docker-library-httpd/tree/0fffbf116b8bdb14f6969ae606766f8f9dd468d1):

$ git rev-parse HEAD
0fffbf116b8bdb14f6969ae606766f8f9dd468d1

$ sudo buildctl build --frontend dockerfile.v0 --local dockerfile=2.4 --local context=2.4 --opt build-arg:SOURCE_DATE_EPOCH=1705159178 --output type=oci,name=build-0,dest=/tmp/build-0.tar,dir=false,rewrite-timestamp=true
[...]
=> exporting to oci image format                                                                                                                                                                                                               4.0s
 => => exporting layers                                                                                                                                                                                                                         0.0s
 => => rewriting layers with source-date-epoch 1705159178 (2024-01-13 15:19:38 +0000 UTC)                                                                                                                                                       3.6s
 => => exporting manifest sha256:37daef8c7bd3c1b46d45b2fd040aa165b17afa85e8f3f845a765696785221c00                                                                                                                                               0.0s
 => => exporting config sha256:467bdfa7f53f9feef496527e163472237e7caca94d02631b074a75c802645619                                                                                                                                                 0.0s
 => => sending tarball

$ sudo buildctl build --frontend dockerfile.v0 --local dockerfile=2.4 --local context=2.4 --opt build-arg:SOURCE_DATE_EPOCH=1705159178 --output type=oci,name=build-1,dest=/tmp/build-1.tar,dir=false,rewrite-timestamp=true --no-cache --secret id=enable-repro-sources-list,source=/dev/null
[...]
=> exporting to oci image format                                                                                                                                                                                                               6.7s 
 => => exporting layers                                                                                                                                                                                                                         2.8s 
 => => rewriting layers with source-date-epoch 1705159178 (2024-01-13 15:19:38 +0000 UTC)                                                                                                                                                       3.5s 
 => => exporting manifest sha256:37daef8c7bd3c1b46d45b2fd040aa165b17afa85e8f3f845a765696785221c00                                                                                                                                               0.0s 
 => => exporting config sha256:467bdfa7f53f9feef496527e163472237e7caca94d02631b074a75c802645619                                                                                                                                                 0.0s 
 => => sending tarball

Used moby/buildkit@a091126 (master) on Ubuntu 23.10 amd64

[AkihiroSuda]

(Note: while snapshot[-cloudflare].debian.org is quite slow, snapshot.ubuntu.com for Ubuntu seems fast enough to adopt as the default)

https://ubuntu.com/blog/ubuntu-snapshots-on-azure-ensuring-predictability-and-consistency-in-cloud-deployments

[AkihiroSuda]

If this PR is still not acceptable, I'd like to open a PR to add the Ubuntu variant with the fast snapshot.ubuntu.com.

[tianon]

Thank you for your contribution ❤️

As noted at docker-library/official-images#16044 (comment), this is not something we're comfortable with at this time -- perhaps at some point in the future we can revisit it.

[AkihiroSuda]

Thank you for your contribution ❤️

As noted at docker-library/official-images#16044 (comment), this is not something we're comfortable with at this time -- perhaps at some point in the future we can revisit it.

Thanks for taking a look, but there might be a confusion.
This PR does not enable the slow snapshot.debian.org for the upstream builds.
Using snapshot.debian.org is opt-in via --secret id=enable-repro-sources-list,source=/dev/null.
This should be only opted in by third-party reproducers.

Could you consider reopening the PR if this sounds good?

[AkihiroSuda]

Alternative:

• Support reproducible builds (except packages) #250

This PR does not contain anything relevant to snapshot.debian.org.
Hope this is easier to merge.