apko_config -> apko_build loses information about the original request #208
Description
Our normal usage of the provider, as codified in terraform-publisher-apko, passes an unresolved config to apko_config, which resolves the package versions and emits the resolved config as an attribute. We then pass that resolved config to apko_build to build the image. We do this mainly to isolate resolution from building, and so we can attest and attach the resolved config to the built image.
However, this means that apko_build produces an image with an /etc/apk/world with fully resolved packages, locking them in place. As far as apko_build is concerned, it was asked to build an image with exactly those packages at those versions, so it did that. The original request -- which we should write to the resulting image's /etc/apk/world -- was lost in the resolving process.
The apko CLI by itself doesn't exhibit this behavior:
$ crane export $(apko publish examples/wolfi-base.yaml ttl.sh/jason) - | tar -tvf - | grep world
wolfi-base
An image produced by apko_config -> apko_build does:
$ crane export cgr.dev/chainguard/wolfi-base - | tar -Oxf - etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r4
ca-certificates-bundle=20230506-r1
glibc-locale-posix=2.38-r8
glibc=2.38-r8
ld-linux=2.38-r8
libcrypt1=2.38-r8
libcrypto3=3.2.0-r0
libssl3=3.2.0-r0
openssl-config=3.2.0-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r7
wolfi-keys=1-r5
zlib=1.3-r2
This matters because the original user's request was not "exactly all these packages at these versions" -- it was "whatever it takes to install wolfi-base please". That unresolved request is what /etc/apk/world communicates.
This is practically a problem when users run apk add in an image built in this way, with a fully locked APK world. Take a months-old wolfi-base image and try to apk add curl
$ docker run --rm -it cgr.dev/chainguard/wolfi-base@sha256:a8c9c2888304e62c133af76f520c9c9e6b3ce6f1a45e3eaa57f6639eb8053c90
# cat /etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r2
ca-certificates-bundle=20230506-r0
glibc-locale-posix=2.38-r5
glibc=2.38-r5
ld-linux=2.38-r5
libcrypt1=2.38-r5
libcrypto3=3.1.4-r0
libssl3=3.1.4-r0
openssl-config=3.1.4-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r6
wolfi-keys=1-r5
zlib=1.3-r1
Note the locked world, with openssl libraries locked at 3.1.4, the latest available at the time the image was built.
Now let's add curl:
# apk add curl
fetch https://packages.wolfi.dev/os/aarch64/APKINDEX.tar.gz
(1/5) Installing libbrotlicommon1 (1.1.0-r1)
(2/5) Installing libbrotlidec1 (1.1.0-r1)
(3/5) Installing libnghttp2-14 (1.58.0-r1)
(4/5) Installing libcurl-openssl4 (8.5.0-r0)
(5/5) Installing curl (8.5.0-r0)
OK: 14 MiB in 19 packages
de3f7a5efe0d:/# cat /etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r2
ca-certificates-bundle=20230506-r0
curl
glibc=2.38-r5
glibc-locale-posix=2.38-r5
ld-linux=2.38-r5
libcrypt1=2.38-r5
libcrypto3=3.1.4-r0
libssl3=3.1.4-r0
openssl-config=3.1.4-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r6
wolfi-keys=1-r5
zlib=1.3-r1
apk add succeeded, but it did the wrong thing, because the curl I just installed needs a newer openssl than the one we locked into the image.
# curl https://google.com
curl: /usr/lib/libssl.so.3: version `OPENSSL_3.2.0' not found (required by /usr/lib/libcurl.so.4)
This could have been avoided if the world hadn't been locked to the set of packages that was appropriate at the time the image was built, but instead reflected the original request.
Unpinning the world and apk upgradeing seems to fix the glitch:
# cat < EOF > /etc/apk/world
apk-tools
busybox
ca-certificates-bundle
curl
glibc
glibc-locale-posix
ld-linux
libcrypt1
libcrypto3
libssl3
openssl-config
wolfi-base
wolfi-baselayout
wolfi-keys
zlib
EOF
# apk upgrade
Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.14.0-r0 -> 2.14.0-r1)
Continuing the upgrade transaction with new apk-tools:
(1/11) Upgrading ca-certificates-bundle (20230506-r0 -> 20230506-r1)
(2/11) Upgrading wolfi-baselayout (20230201-r6 -> 20230201-r7)
(3/11) Upgrading ld-linux (2.38-r5 -> 2.38-r8)
(4/11) Upgrading glibc-locale-posix (2.38-r5 -> 2.38-r8)
(5/11) Upgrading glibc (2.38-r5 -> 2.38-r8)
(6/11) Upgrading openssl-config (3.1.4-r0 -> 3.2.0-r1)
(7/11) Upgrading libcrypto3 (3.1.4-r0 -> 3.2.0-r1)
(8/11) Upgrading libssl3 (3.1.4-r0 -> 3.2.0-r1)
(9/11) Upgrading zlib (1.3-r1 -> 1.3-r3)
(10/11) Upgrading libcrypt1 (2.38-r5 -> 2.38-r8)
(11/11) Upgrading busybox (1.36.1-r2 -> 1.36.1-r4)
Executing glibc-2.38-r8.trigger
Executing busybox-1.36.1-r4.trigger
OK: 15 MiB in 19 packages
# curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
🎉
We also could have made the world just wolfi-base and curl and that would work too -- all the packages in the world come from the original request for the wolfi-base package.
Back to tf-apko:
This locking behavior is not exhibited by apko itself, as demonstrated above with apko publish. It's only an artifact of how we resolve-then-build using tf-apko.
If apko_config's output retained some information about its original request that it could feed to apko_build to populate its unlocked world, we could correctly populate the unlocked world. It may even make sense to attest both the locked package-versions, and the original requested packages (after all TF variables/provider overrides were taken into account), since this could be useful information, and we lose it today.
Another alternative would be to have apko_build produce a locked resolved config output and take the unresolved config as an input.
Instead of
data "apko_config" "this" {
config_contents = file("foo.apko.yaml")
extra_packages = var.extra_packages
default_annotations = var.default_annotations
}
resource "apko_build" "this" {
repo = var.target_repository
config = data.apko_config.this.config
}
resource "cosign_attest" "this" {
predicates {
type = "https://apko.dev/image-configuration"
json = jsonencode(data.apko_config.this.config)
}
}...we'd have:
resource "apko_build" "this" {
repo = var.target_repository
extra_packages = var.extra_packages
default_annotations = var.default_annotations
config_contents = file("foo.apko.yaml") # <-- take the unresolved input
}
resource "cosign_attest" "this" {
predicates {
type = "https://apko.dev/image-configuration"
json = jsonencode(data.apko_build.this.resolved_config) # <-- take the build's resolved config output
}
}(We already attest after building, since attesting depends on check-sbom)