CI: The publish task doesn't work any longer

[larskanis]

... because the cache size is too limited due to changes in Github Actions: https://github.blog/changelog/2025-11-20-github-actions-cache-size-can-now-exceed-10-gb-per-repository/

The cache entries are now immediately deleted when the cache size of 10GB is exceeded. In the past they were deleted after some days only. But our multi-platform docker images need the cache to publish combined ARM and X86_64 images. They are built on native hosts, then stored to the cache and then both caches are combined on ARM hosts to publish the multi-platform images. But the cache is deleted before the publish action runs and then the multi-images can't be built.

[flavorjones]

I was just about to open an issue to start this conversation with you!

See https://github.com/rake-compiler/rake-compiler-dock/actions/runs/20348369065/job/58474748622 for a recent failure. I'm going to build and push 1.11.0.rc1 locally (from my dev machine) for now.

[flavorjones]

Err, I see your notes about requiring ARM hardware in order to build. If we revert to an old version of the image pipeline, will that avoid the cache limit problem?

Or I could try building from a clone that's under my personal account, and just pay for a larger cache (in the short term) until we figure something out.

Do you have any other ideas?

[flavorjones]

Attempting to build this under my own account: https://github.com/flavorjones/rake-compiler-dock/actions/runs/20351170140

If this succeeds, then I'll download all the images, then re-upload them to ghcr.io/rake-compiler/rake-compiler-dock-image

[larskanis]

I think the multi-platform images need too much cache in general, since X86_64 and ARM space is added together. I can build the images locally, because I have both host types available and they work in parallel when using docker buildx with this setup. They build all images in roughly half an hour. So the release is no problem. But automated weekly CI builds would still be nice.

Maybe there is some other storage we could use for the CI build artifacts.

Currently there is some overlapping between the caches. Maybe there's a solution to avoid it.

[flavorjones]

@larskanis If you're able to build locally, would you mind releasing v1.11.0.rc1, then? 🙏

Maybe there is some other storage we could use for the CI build artifacts.

I think in this case paying Github for additional cache might be the cheapest/easiest option? Here's how I've configured my personal account settings:

[larskanis]

Tomorrow morning - I'm too tiered to do a release.

[flavorjones]

It might be interesting to look at the failures here:

https://github.com/flavorjones/rake-compiler-dock/actions/runs/20351170140/job/58479316219

2025-12-18T21:41:36.5361675Z ##[group]Run actions/cache/restore@v4
2025-12-18T21:41:36.5361975Z with:
2025-12-18T21:41:36.5362144Z   path: tmp/build-cache-ARM64
2025-12-18T21:41:36.5362464Z   key: Linux-on-ARM64-arm64-darwin-e8d047792242bc42f8eb32e981d801ae7e91af7b
2025-12-18T21:41:36.5362864Z   restore-keys: Linux-on-ARM64-arm64-darwin
2025-12-18T21:41:36.5363113Z   enableCrossOsArchive: true
2025-12-18T21:41:36.5364334Z   fail-on-cache-miss: true
2025-12-18T21:41:36.5364544Z   lookup-only: false
2025-12-18T21:41:36.5364719Z ##[endgroup]
2025-12-18T21:41:36.8079095Z ##[error]Failed to restore cache entry. Exiting as fail-on-cache-miss is set. Input key: Linux-on-ARM64-arm64-darwin-e8d047792242bc42f8eb32e981d801ae7e91af7b

I not near the cache limit in my personal org, so I suspect there's something else going wrong here.

[flavorjones]

Next idea: Run the pipelines from v1.10.0 to build the v1.11.0.rc1 tag:

https://github.com/rake-compiler/rake-compiler-dock/actions/runs/20352423144

(which is something you can do with workflow_dispatch)

Edit: This didn't work, either. Still running into cache limits somehow.

Edit 2: I figured out how to set up a budget, caching is working again, running the tree-dep pipeline at https://github.com/flavorjones/rake-compiler-dock/actions/runs/20354979001/job/58488420775

[flavorjones]

Fun fact: the publish pipeline requires 13.05 GiB of cache space to build everything.

Lars, it doesn't look like the publish pipeline steps are properly reading from the cache if I re-run the pipeline. See https://github.com/flavorjones/rake-compiler-dock/actions/runs/20358770575/job/58499603765 for an example of a cache hit and full recompute of all layers. I couldn't figure out why.

So I almost got to the point of having this build properly in Github Actions, but I need to get some sleep. Sorry I couldn't get it done tonight.

[flavorjones]

@larskanis Do you think you'll be able to cut a release today? If not please let me know. 🙏

[larskanis]

@larskanis Do you think you'll be able to cut a release today? If not please let me know. 🙏

Yes, I'm on it right now!

[larskanis]

Done, 1.11.0.rc1 is released. It took a while to push all the images through my 10MBit upstream wire.

[flavorjones]

@larskanis Thank you so much!

[flavorjones]

@larskanis Do you have any ideas on how to fix this? If not, then should we move this to another github org where we have control over github billing and cache sizes?

[flavorjones]

Not sure if you're around today, so I'll just say that my current plan is to build the images under my fork using the existing main pipeline, and then manually push those images to https://github.com/rake-compiler/rake-compiler-dock/pkgs/container/rake-compiler-dock-image. If you've got other ideas, let me know!