Issues with continously downloading bellsoft-liberica-vm-core-openjdk for every build

[Azbesciak]

Hi, I have a spring boot apps, which are using packeto to build native images.
The config in gradle looks like

bootBuildImage {
            builder = "paketobuildpacks/builder:tiny"
            pullPolicy.set(PullPolicy.IF_NOT_PRESENT)
            imageName = "docker.io/library/${project.name}:${project.version == "unspecified" ? "latest" : project.version}-aot"
            buildCache {
                it.volume {
                    it.name = "pack-cache-route-service-aot${System.getenv("AOT_IMG_CACHE_VER") ?: ""}.build"
                }
            }
            environment = [
                    "BP_NATIVE_IMAGE": "true"
            ]
            // https://hub.docker.com/r/bellsoft/liberica-native-image-kit-container/tags
//            buildpacks = [
//                    "paketo-buildpacks/ca-certificates",
//                    "paketo-buildpacks/bellsoft-liberica",
//                    "paketo-buildpacks/syft",
//                    "paketo-buildpacks/executable-jar",
//                    "paketo-buildpacks/spring-boot",
//                    "paketo-buildpacks/native-image",
//                    "docker://docker.io/paketobuildpacks/builder:tiny",
//                    "docker://bellsoft/liberica-native-image-kit-container:jdk-17-nik-22.3.2-musl"
//            ]
        }

Both AOT_IMG_CACHE_VER and project.version are constant, and still I see in logs

BellSoft Liberica NIK 17.0.7: Contributing to layer
[creator]         Downloading from https://download.bell-sw.com/vm/22.3.2/bellsoft-liberica-vm-core-openjdk17.0.7+7-22.3.2+1-linux-amd64.tar.gz
[creator]         Verifying checksum
[creator]         Expanding to /layers/paketo-buildpacks_bellsoft-liberica/native-image-svm

all the time. I am coming there from paketo-buildpacks/java#86, I have tried to use squid for caching, but none tutorials worked for me for docker based version.

I just wonder - is it not possible to just use bellsoft/liberica-native-image-kit-container:jdk-17-nik-22.3.2-musl instead of fetching the tar.gz? There is a docker env anyway, so why not just call docker pull? As you see I have made some tries, but this is probably deeply wrong.

Any help is appreciated, I am making tons of the carbon footprint :)

[dmikusa]

Both AOT_IMG_CACHE_VER and project.version are constant, and still I see in logs

Good to know, that's a common cause of this issue. You definitely should not see it download every time. If we look at the debug logs for the buildpacks it'll give us some more clues.

Please include "BP_DEBUG": "true" in your environment block and include two full sets of build logs. The first should be from a clean build with no previous build caching. The second should be a rebuild, no changes to code, just immediately rebuild after the first time. Attach those here or link to them.

Aside from that, I would strongly suggest you repeat the previous test using a pack build -v -e BP_DEBUG=true ... (same options). i.e. build a fresh app with no previous cache, then immediately rebuild and check if the problem occurs here as well. Also, attach or link to the output of both builds. The reason being is that pack will output more information about the lifecycle (the -v arg triggers this), and I do not believe there's an equivalent option with Spring Boot build tool.

I have tried to use squid for caching, but none tutorials worked for me for docker based version.

You could cache at the network layer, but that's probably overkill unless you've got a whole team of people.

Other solutions:

• Use binding-tool. It will pre-download binaries and configure dependency-mappings for you such that buildpacks will use file:// URLs to fetch dependencies from its pre-downloaded binaries. This is a great option for a single developer.

• Package your own copy of buildpacks using offline mode. This can be a bit of work, because it requires you to build your own copy of the buildpacks, but it also gives you a lot of control in terms of what dependencies you include and when you upgrade buildpacks. It also allows you to bundle dependencies into the container, so that they are effectively cached all the time. It's probably overkill unless you're working with a team of people all doing builds.

I just wonder - is it not possible to just use bellsoft/liberica-native-image-kit-container:jdk-17-nik-22.3.2-musl instead of fetching the tar.gz? There is a docker env anyway, so why not just call docker pull? As you see I have made some tries, but this is probably deeply wrong.

That's not how buildpacks work. Buildpacks manage dependencies in a way that the base image/layer does not include the dependencies. This provides the advantage that you can rebase and update images without actually rebuilding, a feature that is very convenient if you need to update many images after a security issue in your base image.

[dmikusa]

Longer term, you can take a look at paketo-buildpacks/rfcs#287 to see what we're planning. Feedback on that RFC is also appreciated.

[Azbesciak]

@dmikusa

We're just downloading large files, so you'd need your proxy to cache them.

You are downloading large files with HTTPS what just requires Man in the middle ;)

[Azbesciak]

You know, I would never say that such a simple thing as replacing the source for some dependency would require so much effort... IMO that is just a convenient configuration stuff. Also notice once again, that I am using bootBuildImage from spring, it also has limited configuration.

[dmikusa]

The complexity comes from security and integrity requirements for buildpacks. There is a need/demand for secure software delivery.

• Using HTTPS is a requirement to prevent software (and associated metadata) from being MITM'd. This is table stakes in package managers (look at the variety of Linux Package Managers) or software update tools. It helps to guarantee that the software you request is the software you receive.
• Similarly, we do not allow the software deployed through the buildpack to be arbitrarily replaced. We only allow dependencies to be replaced if the sha256 hash of the original dependency matches the sha256 hash of the overridden dependency, which again, helps to ensure that the expected software has been installed. This is the rationale behind the dependency mapping system. It allows secure replacement of dependencies.

In some cases, you need more control or more flexibility. For example, you want or need to be able to dictate exactly what versions are bundled with the buildpack. Another common case is needing to bundle dependencies into the buildpack itself for caching or working in air-gapped environments. The process above is more involved, but it is also only suggested for more advanced use cases.

[Azbesciak]

But @dmikusa , this is my problem what I have in my image. Why should you care about it, if you provided defaults? What if I have my internal repo over http in my private net, or I want to chage the way dependencies are fetched? Or I have it on some cloud, which requires auth?

[dmikusa]

Oh, one other question, What is your container system? Docker? Podman? CI?

[Azbesciak]

Docker, on windows, and I am calling it with bootBuildImage, not pack or something, that is also a problem on my side because it is hidden

[dmikusa]

I would also suggest updating to the latest buildpacks. There have been a couple of issues resolved recently with dependency download caching. You're on Paketo Buildpack for BellSoft Liberica 10.2.4 which is 5 versions behind. I believe you're missing one of the fixes.

Again, the behavior you're seeing shouldn't happen. Network caching is great, offline buildpacks, and dependency mappings are not generally necessary because there should be a default level of caching that's not happening here. The build should download once the first time you build successfully for the given image name. Then rebuilds for the same image name should cache and reuse those downloads.

Please try with the latest buildpack versions and let me know how that goes. If you're still not seeing the expected behavior give me another set of builds (first build & second build, delete your docker volumes before the first build to clear the cache). Thanks

[Azbesciak]

I will try, thank you, but first I need to figure out how to replate that it my spring, which is also not the newest one because the newest introduced new bugs with aot...

[dmikusa]

You have pullPolicy.set(PullPolicy.IF_NOT_PRESENT) set so it won't download the images if they already exist in your Docker. If you change that to ALWAYS or if you just delete the images from Docker then build again, you should get the latest builder.