Raspberry Pi: Latest kernel grew larger (again) than U-Boot allows

[LordMZTE]

Describe the bug

The U-Boot bootloader NixOS ships with on aarch64 SD card images limits kernel size to 71827456 bytes (68.5 MiB) as per this patch. This limit is very easy to hit.

Steps To Reproduce

Steps to reproduce the behavior:

1. Grab a Raspberry Pi (3B+ in my case)
2. Add this configuration:

boot.kernelPackages = (import nixpkgs {
  system = "x86_64-linux";
  crossSystem = "aarch64-linux";
}).linuxPackages_latest;

boot.kernelPatches = [{
  name = "enable-gpio-poweroff";
  patch = null;
  extraStructuredConfig.POWER_RESET_GPIO = lib.kernel.yes;
}];

Note: the kernel patch may not be required to hit the limit, this additional kernel module is not directly related to the issue.

My kernel image (version 6.8.1) ended up being just over the limit at 68.83 MiB.

Nixpkgs commit this was reproduced on: 57e6b3a
3. Build the system. The bootloader will fall back to the last generation where the kernel was within the size limit.

Expected behavior

The bootloader would either be able to dynamically allocate the address space in order to allow any size of kernel image (not sure if this is possible with U-Boot) or the maximum should be adjusted to give some more room.

Additional context

Add any other context about the problem here.

Notify maintainers

@bartsch
@dezgeg
@samueldr
@lopsided98

Metadata

 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.22, NixOS, 24.05 (Uakari), 24.05.20240325.57e6b3a`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixos-24.05pre603596.2726f127c15a"`
 - nixpkgs: `/nix/store/161i7cxz0w0raqmrj88z86cj66babp9z-source`

[samueldr]

Reworded the title to show that (1) it's device-specific and (2) a limitation within U-Boot...

... and it is a limitation we were already having to work around.

[tomfitzhenry]

I'm experiencing the same error ("ERROR: RD image overlaps OS image") on boot with my Pine64 RockPro64 running linuxPackages_6_6 on nixos-unstable, and Tow Boot .

It manifested itself as u-boot falling back to an old boot entry, so it took me a while to notice it (on reboots, I would boot into older generations).

I found a NanoPI4S user, also a NixOS user, that has the same boot error: friendlyarm/uboot-rockchip#11

What are our options?

1. Figure out why the kernel image has gotten larger, and try to reduce it
2. Apply https://github.com/friendlyarm/uboot-rockchip/pull/11/files to upstream u-boot, Tow-Boot, and/or NixOS's u-boot configs?
3. Anything else ... ?

When I have time, I'll work on figuring out which NixOS commit resulted in a larger kernel.

[tomfitzhenry]

When I have time, I'll work on figuring out which NixOS commit resulted in a larger kernel.

Brief findings so far:

$ cd /boot/nixos
$ ls -lh *Image
-rwxr-xr-x 1 root root 67M Apr 28 10:37 imgi3sal09fsdrqch0s3pfxsp85slr8z-linux-aarch64-unknown-linux-gnu-6.6.28-Image
-rwxr-xr-x 1 root root 62M Feb 16 08:38 wbz58n20ywjg3xqxj25j57mp51vqmnfs-linux-aarch64-unknown-linux-gnu-6.6.16-Image

I apply a local kernel patch (for reliable PCIe booting), so you won't file those exact files in Hydra.

[nektro]

I reproduced this as well on my pinebook pro after upgrading from 23.11 to 24.05

[tomfitzhenry]

For RK3399 devices such as RockPro64 and Pinebook Pro, per Tow-Boot/U-Boot#3 , Tow-Boot/Tow-Boot#280 and https://github.com/Tow-Boot/Tow-Boot/releases/tag/release-2023.07-007 , this looks to have been fixed in Tow-Boot 2023.07-007.

Thanks to @samueldr and @artemist for fixing this!

I'll leave this open, since the issue still (?) impacts Raspberry Pi.