build: add --source-date-epoch and --rewrite-timestamp flags

Use $SOURCE_DATE_EPOCH as the default for the --source-date-epoch flag
to the "build" CLI.

When a source-date-epoch is set, we'll use it when writing new history
entries, force timestamps in data written for --output to the specified
timestamp, and populate a "SOURCE_DATE_EPOCH" ARG that we treat as
always being set, and which we don't complain about being left unused.
By default, this will not affect timestamps in newly-added layers.

Add a --rewrite-timestamp flag, which "clamps" timestamps in newly-added
layers to not be later than the --source-date-epoch value if the
--source-date-epoch flag is set, but has no effect otherwise.

Signed-off-by: Nalin Dahyabhai <[email protected]>

Author: nalind

define/build.go

@@ -295,9 +295,23 @@ type BuildOptions struct {
 	SignBy string
 	// Architecture specifies the target architecture of the image to be built.
 	Architecture string
-	// Timestamp sets the created timestamp to the specified time, allowing
-	// for deterministic, content-addressable builds.
+	// Timestamp specifies a timestamp to use for the image's created-on
+	// date, the corresponding field in new history entries, the timestamps
+	// to set on contents in new layer diffs, and the timestamps to set on
+	// contents written as specified in the BuildOutput field.  If left
+	// unset, the current time is used for the configuration and manifest,
+	// and layer contents are recorded as-is.
 	Timestamp *time.Time
+	// SourceDateEpoch specifies a timestamp to use for the image's
+	// created-on date and the corrsponding field in new history entries,
+	// and any content written as specified in the BuildOutput field.  If
+	// left unset, the current time is used for the configuration and
+	// manifest, and layer and BuildOutput contents retain their original
+	// timestamps.
+	SourceDateEpoch *time.Time
+	// RewriteTimestamp, if set, forces timestamps in generated layers to
+	// not be later than the SourceDateEpoch, if it is also set.
+	RewriteTimestamp bool
 	// OS is the specifies the operating system of the image to be built.
 	OS string
 	// MaxPullPushRetries is the maximum number of attempts we'll make to pull or push any one

docs/buildah-build.1.md

@@ -715,24 +715,34 @@ Windows base images, so using this option is usually unnecessary.
 
 **--output**, **-o**=""
 
-Output destination (format: type=local,dest=path)
+Additional output (format: type=local,dest=path)
 
-The --output (or -o) option extends the default behavior of building a container image by allowing users to export the contents of the image as files on the local filesystem, which can be useful for generating local binaries, code generation, etc.
+The --output (or -o) option supplements the default behavior of building a
+container image by allowing users to export the image's contents as files on
+the local filesystem, which can be useful for generating local binaries, code
+generation, etc.
 
-The value for --output is a comma-separated sequence of key=value pairs, defining the output type and options.
+The value for --output is a comma-separated sequence of key=value pairs,
+defining the output type and options.
 
 Supported _keys_ are:
-- **dest**: Destination path for exported output. Valid value is absolute or relative path, `-` means the standard output.
-- **type**: Defines the type of output to be used. Valid values is documented below.
+ **dest**: Destination for exported output. Can be set to `-` to indicate standard output, or to an absolute or relative path.
+ **type**: Defines the type of output to be written. Must be one of the values listed below.
 
 Valid _type_ values are:
-- **local**: write the resulting build files to a directory on the client-side.
-- **tar**: write the resulting files as a single tarball (.tar).
+ **local**: write the resulting build files to a directory on the client-side.
+ **tar**: write the resulting files as a single tarball (.tar).
 
-If no type is specified, the value defaults to **local**.
-Alternatively, instead of a comma-separated sequence, the value of **--output** can be just a destination (in the `**dest**` format) (e.g. `--output some-path`, `--output -`) where `--output some-path` is treated as if **type=local** and `--output -` is treated as if **type=tar**.
+Alternatively, instead of a comma-separated sequence, the value of **--output**
+can be just the destination (in the `**dest**` format) (e.g. `--output
+some-path`, `--output -`), and the **type** will be inferred to be **tar** if
+the output destination is `-`, and **local** otherwise.
 
-Note: The **--tag** option can also be used to change the file image format to supported `containers-transports(5)`.
+Timestamps on the output contents will be set to exactly match the value
+specified using the **--timestamp** flag, or to exactly match the value
+specified for the **--source-date-epoch** flag, if either are specified.
+
+Note that the **--tag** option can also be used to write the image to any location described by `containers-transports(5)`.
 
 **--pid** *how*
 
@@ -802,6 +812,13 @@ Duration of delay between retry attempts in case of failure when performing push
 
 Defaults to `2s`.
 
+**--rewrite-timestamp**
+
+When generating new layers for the image, ensure that no newly added content
+bears a timestamp later than the value used by the **--source-date-epoch**
+flag, if one was provided, by replacing any timestamps which are later than
+that value, with that value.
+
 **--rm** *bool-value*
 
 Remove intermediate containers after a successful build (default true).
@@ -970,6 +987,31 @@ Sign the built image using the GPG key that matches the specified fingerprint.
 
 Skip stages in multi-stage builds which don't affect the target stage. (Default is `true`).
 
+**--source-date-epoch** *seconds*
+
+Set the "created" timestamp for the built image to this number of seconds since
+the epoch (Unix time 0, i.e., 00:00:00 UTC on 1 January 1970) (default is to
+use the value set in the `SOURCE_DATE_EPOCH` environment variable, or the
+current time if it is not set).
+
+The "created" timestamp is written into the image's configuration and manifest
+when the image is committed, so running the same build two different times
+will ordinarily produce images with different sha256 hashes, even if no other
+changes were made to the Containerfile and build context.
+
+When this flag is set, a `SOURCE_DATE_EPOCH` build arg will provide its value
+for a stage in which it is declared.
+
+When this flag is set, the image configuration's "created" timestamp is always
+set to the time specified, which should allow for identical images to be built
+at different times using the same set of inputs.
+
+When this flag is set, output written as specified to the **--output** flag
+will bear exactly the specified timestamp.
+
+Conflicts with the similar **--timestamp** flag, which also sets its specified
+time on the contents of new layers.
+
 **--squash**
 
 Squash all layers, including those from base image(s), into one single layer. (Default is false).
@@ -1013,10 +1055,25 @@ Commands after the target stage will be skipped.
 
 **--timestamp** *seconds*
 
-Set the create timestamp to seconds since epoch to allow for deterministic builds (defaults to current time).
-By default, the created timestamp is changed and written into the image manifest with every commit,
-causing the image's sha256 hash to be different even if the sources are exactly the same otherwise.
-When --timestamp is set, the created timestamp is always set to the time specified and therefore not changed, allowing the image's sha256 to remain the same. All files committed to the layers of the image will be created with the timestamp.
+Set the "created" timestamp for the built image to this number of seconds since
+the epoch (Unix time 0, i.e., 00:00:00 UTC on 1 January 1970) (defaults to
+current time).
+
+The "created" timestamp is written into the image's configuration and manifest
+when the image is committed, so running the same build two different times
+will ordinarily produce images with different sha256 hashes, even if no other
+changes were made to the Containerfile and build context.
+
+When --timestamp is set, the "created" timestamp is always set to the time
+specified, which should allow for identical images to be built at different
+times using the same set of inputs.
+
+When --timestamp is set, all content in layers created as part of the build,
+and output written as specified to the **--output** flag, will also bear this
+same timestamp.
+
+Conflicts with the similar **--source-date-epoch** flag, which by default does
+not affect the timestamps of layer contents.
 
 **--tls-verify** *bool-value*
 

docs/buildah-commit.1.md

@@ -318,6 +318,9 @@ When --source-date-epoch is set, the "created" timestamp is always set to the ti
 specified, which should allow for identical images to be committed at different
 times.
 
+Conflicts with the similar **--timestamp** flag, which also sets its specified
+time on layer contents.
+
 **--squash**
 
 Squash all of the new image's layers (including those inherited from a base image) into a single new layer.
@@ -338,6 +341,9 @@ specified, which should allow for identical images to be committed at different
 times.  All content in the new layer added as part of the image will also bear
 this timestamp.
 
+Conflicts with the similar **--source-date-epoch** flag, which by default does
+not affect the timestamps of layer contents.
+
 **--tls-verify** *bool-value*
 
 Require HTTPS and verification of certificates when talking to container registries (defaults to true).  TLS verification cannot be used when talking to an insecure registry.

image.go

@@ -52,9 +52,10 @@ const (
 // control whether various information like the like setuid and setgid bits and
 // xattrs are preserved when extracting file system objects.
 type ExtractRootfsOptions struct {
-	StripSetuidBit bool // strip the setuid bit off of items being extracted.
-	StripSetgidBit bool // strip the setgid bit off of items being extracted.
-	StripXattrs    bool // don't record extended attributes of items being extracted.
+	StripSetuidBit bool       // strip the setuid bit off of items being extracted.
+	StripSetgidBit bool       // strip the setgid bit off of items being extracted.
+	StripXattrs    bool       // don't record extended attributes of items being extracted.
+	ForceTimestamp *time.Time // force timestamps in output content
 }
 
 type containerImageRef struct {
@@ -226,13 +227,14 @@ func (i *containerImageRef) extractRootfs(opts ExtractRootfsOptions) (io.ReadClo
 	pipeReader, pipeWriter := io.Pipe()
 	errChan := make(chan error, 1)
 	go func() {
+		defer pipeWriter.Close()
 		defer close(errChan)
 		if len(i.extraImageContent) > 0 {
 			// Abuse the tar format and _prepend_ the synthesized
 			// data items to the archive we'll get from
 			// copier.Get(), in a way that looks right to a reader
 			// as long as we DON'T Close() the tar Writer.
-			filename, _, _, err := i.makeExtraImageContentDiff(false)
+			filename, _, _, err := i.makeExtraImageContentDiff(false, opts.ForceTimestamp)
 			if err != nil {
 				errChan <- fmt.Errorf("creating part of archive with extra content: %w", err)
 				return
@@ -257,10 +259,10 @@ func (i *containerImageRef) extractRootfs(opts ExtractRootfsOptions) (io.ReadClo
 			StripSetuidBit: opts.StripSetuidBit,
 			StripSetgidBit: opts.StripSetgidBit,
 			StripXattrs:    opts.StripXattrs,
+			Timestamp:      opts.ForceTimestamp,
 		}
 		err := copier.Get(mountPoint, mountPoint, copierOptions, []string{"."}, pipeWriter)
 		errChan <- err
-		pipeWriter.Close()
 	}()
 	return ioutils.NewReadCloserWrapper(pipeReader, func() error {
 		if err = pipeReader.Close(); err != nil {
@@ -849,7 +851,7 @@ func (i *containerImageRef) NewImageSource(_ context.Context, _ *types.SystemCon
 			layerUncompressedSize = apiLayers[apiLayerIndex].size
 		} else if layerID == synthesizedLayerID {
 			// layer diff consisting of extra files to synthesize into a layer
-			diffFilename, digest, size, err := i.makeExtraImageContentDiff(true)
+			diffFilename, digest, size, err := i.makeExtraImageContentDiff(true, nil)
 			if err != nil {
 				return nil, fmt.Errorf("unable to generate layer for additional content: %w", err)
 			}
@@ -1152,7 +1154,7 @@ func (i *containerImageSource) GetBlob(_ context.Context, blob types.BlobInfo, _
 // makeExtraImageContentDiff creates an archive file containing the contents of
 // files named in i.extraImageContent.  The footer that marks the end of the
 // archive may be omitted.
-func (i *containerImageRef) makeExtraImageContentDiff(includeFooter bool) (_ string, _ digest.Digest, _ int64, retErr error) {
+func (i *containerImageRef) makeExtraImageContentDiff(includeFooter bool, timestamp *time.Time) (_ string, _ digest.Digest, _ int64, retErr error) {
 	cdir, err := i.store.ContainerDirectory(i.containerID)
 	if err != nil {
 		return "", "", -1, err
@@ -1170,9 +1172,12 @@ func (i *containerImageRef) makeExtraImageContentDiff(includeFooter bool) (_ str
 	digester := digest.Canonical.Digester()
 	counter := ioutils.NewWriteCounter(digester.Hash())
 	tw := tar.NewWriter(io.MultiWriter(diff, counter))
-	created := time.Now()
-	if i.created != nil {
-		created = *i.created
+	if timestamp == nil {
+		now := time.Now()
+		timestamp = &now
+		if i.created != nil {
+			timestamp = i.created
+		}
 	}
 	for path, contents := range i.extraImageContent {
 		if err := func() error {
@@ -1189,7 +1194,7 @@ func (i *containerImageRef) makeExtraImageContentDiff(includeFooter bool) (_ str
 				Name:     path,
 				Typeflag: tar.TypeReg,
 				Mode:     0o644,
-				ModTime:  created,
+				ModTime:  *timestamp,
 				Size:     st.Size(),
 			}); err != nil {
 				return err

imagebuildah/build.go

@@ -21,6 +21,7 @@ import (
 	"github.com/containerd/platforms"
 	"github.com/containers/buildah"
 	"github.com/containers/buildah/define"
+	"github.com/containers/buildah/internal"
 	internalUtil "github.com/containers/buildah/internal/util"
 	"github.com/containers/buildah/pkg/parse"
 	"github.com/containers/buildah/util"
@@ -259,6 +260,16 @@ func BuildDockerfiles(ctx context.Context, store storage.Store, options define.B
 		}
 		// Deep copy args to prevent concurrent read/writes over Args.
 		platformOptions.Args = maps.Clone(options.Args)
+
+		if options.SourceDateEpoch != nil {
+			if options.Timestamp != nil {
+				return "", nil, errors.New("timestamp and source-date-epoch would be ambiguous if allowed together")
+			}
+			if _, alreadySet := platformOptions.Args[internal.SourceDateEpochName]; !alreadySet {
+				platformOptions.Args[internal.SourceDateEpochName] = fmt.Sprintf("%d", options.SourceDateEpoch.Unix())
+			}
+		}
+
 		builds.Go(func() error {
 			loggerPerPlatform := logger
 			if platformOptions.LogFile != "" && platformOptions.LogSplitByPlatform {

imagebuildah/executor.go

@@ -14,6 +14,7 @@ import (
 
 	"github.com/containers/buildah"
 	"github.com/containers/buildah/define"
+	"github.com/containers/buildah/internal"
 	internalUtil "github.com/containers/buildah/internal/util"
 	"github.com/containers/buildah/pkg/parse"
 	"github.com/containers/buildah/pkg/sshagent"
@@ -43,18 +44,19 @@ import (
 // instruction in the Dockerfile, since that's usually an indication of a user
 // error, but for these values we make exceptions and ignore them.
 var builtinAllowedBuildArgs = map[string]struct{}{
-	"HTTP_PROXY":     {},
-	"http_proxy":     {},
-	"HTTPS_PROXY":    {},
-	"https_proxy":    {},
-	"FTP_PROXY":      {},
-	"ftp_proxy":      {},
-	"NO_PROXY":       {},
-	"no_proxy":       {},
-	"TARGETARCH":     {},
-	"TARGETOS":       {},
-	"TARGETPLATFORM": {},
-	"TARGETVARIANT":  {},
+	"HTTP_PROXY":                 {},
+	"http_proxy":                 {},
+	"HTTPS_PROXY":                {},
+	"https_proxy":                {},
+	"FTP_PROXY":                  {},
+	"ftp_proxy":                  {},
+	"NO_PROXY":                   {},
+	"no_proxy":                   {},
+	"TARGETARCH":                 {},
+	"TARGETOS":                   {},
+	"TARGETPLATFORM":             {},
+	"TARGETVARIANT":              {},
+	internal.SourceDateEpochName: {},
 }
 
 // Executor is a buildah-based implementation of the imagebuilder.Executor
@@ -164,6 +166,8 @@ type Executor struct {
 	compatVolumes                           types.OptionalBool
 	compatScratchConfig                     types.OptionalBool
 	noPivotRoot                             bool
+	sourceDateEpoch                         *time.Time
+	rewriteTimestamp                        bool
 }
 
 type imageTypeAndHistoryAndDiffIDs struct {
@@ -330,6 +334,8 @@ func newExecutor(logger *logrus.Logger, logPrefix string, store storage.Store, o
 		compatVolumes:                           options.CompatVolumes,
 		compatScratchConfig:                     options.CompatScratchConfig,
 		noPivotRoot:                             options.NoPivotRoot,
+		sourceDateEpoch:                         options.SourceDateEpoch,
+		rewriteTimestamp:                        options.RewriteTimestamp,
 	}
 	if exec.err == nil {
 		exec.err = os.Stderr

imagebuildah/stage_executor.go

@@ -2457,6 +2457,8 @@ func (s *StageExecutor) commit(ctx context.Context, createdBy string, emptyLayer
 		HistoryTimestamp:      s.executor.timestamp,
 		Manifest:              s.executor.manifest,
 		CompatSetParent:       s.executor.compatSetParent,
+		SourceDateEpoch:       s.executor.sourceDateEpoch,
+		RewriteTimestamp:      s.executor.rewriteTimestamp,
 	}
 	if finalInstruction {
 		options.ConfidentialWorkloadOptions = s.executor.confidentialWorkload
@@ -2478,7 +2480,13 @@ func (s *StageExecutor) commit(ctx context.Context, createdBy string, emptyLayer
 }
 
 func (s *StageExecutor) generateBuildOutput(buildOutputOpts define.BuildOutputOption) error {
-	extractRootfsOpts := buildah.ExtractRootfsOptions{}
+	forceTimestamp := s.executor.timestamp
+	if s.executor.sourceDateEpoch != nil {
+		forceTimestamp = s.executor.sourceDateEpoch
+	}
+	extractRootfsOpts := buildah.ExtractRootfsOptions{
+		ForceTimestamp: forceTimestamp,
+	}
 	if unshare.IsRootless() {
 		// In order to maintain as much parity as possible
 		// with buildkit's version of --output and to avoid
@@ -2492,7 +2500,11 @@ func (s *StageExecutor) generateBuildOutput(buildOutputOpts define.BuildOutputOp
 		extractRootfsOpts.StripSetgidBit = true
 		extractRootfsOpts.StripXattrs = true
 	}
-	rc, errChan, err := s.builder.ExtractRootfs(buildah.CommitOptions{}, extractRootfsOpts)
+	rc, errChan, err := s.builder.ExtractRootfs(buildah.CommitOptions{
+		HistoryTimestamp: s.executor.timestamp,
+		SourceDateEpoch:  s.executor.sourceDateEpoch,
+		RewriteTimestamp: s.executor.rewriteTimestamp,
+	}, extractRootfsOpts)
 	if err != nil {
 		return fmt.Errorf("failed to extract rootfs from given container image: %w", err)
 	}

internal/types.go

@@ -6,8 +6,10 @@ const (
 	// external items which are downloaded for a build, typically a tarball
 	// being used as an additional build context.
 	BuildahExternalArtifactsDir = "buildah-external-artifacts"
-	// SourceDateEpochName is the name of the SOURCE_DATE_EPOCH environment
-	// variable when it's read from the environment by our main().
+	// SourceDateEpochName is both the name of the SOURCE_DATE_EPOCH
+	// environment variable and the built-in ARG that carries its value,
+	// whether it's read from the environment by our main(), or passed in
+	// via CLI or API flags.
 	SourceDateEpochName = "SOURCE_DATE_EPOCH"
 )