Segmentation fault in dracut --sysroot due to recursive kernel module installs

[rcorujo]

Describe the bug
A clear and concise description of what the error is.

I am running SLES15SP6 on the host and have a RHEL 8.10 chroot environment for which I would like to create an initramfs.img.

Run this command:

dracut --debug --sysroot /opt/clmgr/image/images/rhel8.10 --kver 4.18.0-553.el8_10.x86_64 --kmoddir /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64 --drivers "usb-storage" --force /tmp/my-initramfs.img

The debug output shows that there is a recursive loop that leads to a segmentation fault.

dracut-install: Handle module 'usb_storage'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz'
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz')
dracut-install: dest dir '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage' does not exist
dracut-install: dracut_install('/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage')
dracut-install: dest dir '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb' does not exist
dracut-install: dracut_install('/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb')
dracut-install: mkdir '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb'
dracut-install: mkdir '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage'
dracut-install: dracut_install ret = 0
dracut-install: cp '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz'
dracut-install: dracut_install ret = 0
dracut-install: dracut_install 'usb_storage' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz')
dracut-install: dracut_install ret = 0
dracut-install: cp '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' '/opt/clmgr/image/images/rhel8.10/var/tmp/dracut.cHd7Tg/initramfs/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz'
dracut-install: dracut_install ret = 0
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz')
...
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz' OK
dracut-install: dracut_install('/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz', '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz')
dracut-install: hash hit items for '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz'
dracut-install: dracut_install '/opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' '/lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz' OK
/opt/clmgr/image/images/rhel8.10/usr/lib/dracut/dracut-init.sh: line 999: 1654880 Segmentation fault      (core dumped) $DRACUT_INSTALL ${initdir:+-D "$initdir"} ${loginstall:+-L "$loginstall"} ${hostonly:+-H} ${omit_drivers:+-N "$omit_drivers"} ${srcmods:+--kerneldir "$srcmods"} ${_optional:+-o} ${_silent:+--silent} -m "$@"

Here are the dependencies in the chroot environment. I'm not sure why the debug output shows that the uas kernel module is being installed, since the usb-storage kernel module does not have a dependency on it. It is the uas kernel module that has a dependency on usb-storage, but I didn't request for the uas kernel module to be installed.

# grep usb-storage /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64/modules.dep
kernel/drivers/usb/storage/uas.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/usb-storage.ko.xz:
kernel/drivers/usb/storage/ums-alauda.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-cypress.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-datafab.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-eneub6250.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-freecom.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-isd200.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-jumpshot.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-karma.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-onetouch.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-realtek.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-sddr09.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-sddr55.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz
kernel/drivers/usb/storage/ums-usbat.ko.xz: kernel/drivers/usb/storage/usb-storage.ko.xz

Distribution used
Which distribution was this behaviour seen in?

SLES15SP6 in the host and RHEL 8.10 in the chroot environment

Dracut version
Which dracut version was this behaviour seen in?

# dracut --version
dracut 059+suse.521.g8412a1c0-150600.1.3

Init system
Which init system is being used?

# ps -p 1 -o comm=
systemd

To Reproduce
Steps or code to reproduce the behavior.

See description of the bug.

Expected behavior
A clear and concise description of what you expected to happen.

No recursive loop and no segmentation fault.

Additional context
Add any other context you like about the problem here.

[rcorujo]

Update: If I omit the uas driver, there are no more recursive module installs and no segmentation fault.

dracut --debug --sysroot /opt/clmgr/image/images/rhel8.10 --kver 4.18.0-553.el8_10.x86_64 --kmoddir /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64 --drivers "usb-storage" --omit-drivers "uas" --force /tmp/my-initramfs.img

However, if I specify that I want the uas driver included, or both the uas and usb-storage drivers included, then the recursion and the segmentation fault is back.

dracut --debug --sysroot /opt/clmgr/image/images/rhel8.10 --kver 4.18.0-553.el8_10.x86_64 --kmoddir /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64 --drivers "uas" --force /tmp/my-initramfs.img

dracut --debug --sysroot /opt/clmgr/image/images/rhel8.10 --kver 4.18.0-553.el8_10.x86_64 --kmoddir /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64 --drivers "uas usb-storage" --force /tmp/my-initramfs.img

[jozzsi]

CC @mwilck and @aafeijoo-suse for SLES

[rcorujo]

The problem does not occur if I explicitly chroot into the directory where the OS image is, and then run dracut without the --sysroot option.

chroot /opt/clmgr/image/images/rhel8.10 dracut --debug --kver 4.18.0-553.el8_10.x86_64 --kmoddir /lib/modules/4.18.0-553.el8_10.x86_64 --drivers "usb-storage" --force /tmp/my-initramfs.img

# lsinitrd /tmp/my-initramfs.img | grep "\.ko"
-rw-r--r--   1 root     root        67608 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/drm_kms_helper.ko.xz
-rw-r--r--   1 root     root       187948 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/drm.ko.xz
-rw-r--r--   1 root     root         6856 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/drm_shmem_helper.ko.xz
-rw-r--r--   1 root     root        21016 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/gpu/drm/mgag200/mgag200.ko.xz
-rw-r--r--   1 root     root         5168 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/i2c/algos/i2c-algo-bit.ko.xz
-rw-r--r--   1 root     root        20884 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/scsi/sg.ko.xz
-rw-r--r--   1 root     root        11308 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/uas.ko.xz
-rw-r--r--   1 root     root        29428 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz
-rw-r--r--   1 root     root         3312 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/video/fbdev/core/syscopyarea.ko.xz
-rw-r--r--   1 root     root         3684 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/video/fbdev/core/sysfillrect.ko.xz
-rw-r--r--   1 root     root         2804 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/drivers/video/fbdev/core/sysimgblt.ko.xz
-rw-r--r--   1 root     root         2472 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/net/802/stp.ko.xz
-rw-r--r--   1 root     root       113020 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/net/bridge/bridge.ko.xz
-rw-r--r--   1 root     root        10996 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/net/bridge/br_netfilter.ko.xz
-rw-r--r--   1 root     root         4784 Jan 15  2024 lib/modules/4.18.0-553.el8_10.x86_64/kernel/net/llc/llc.ko.xz

It seems that the problem may be related to having to specify the full path to the kernel modules when the --sysroot option is used. I think the leading /opt/clmgr/image/images/rhel8.10 is tripping it up.

--kmoddir /opt/clmgr/image/images/rhel8.10/lib/modules/4.18.0-553.el8_10.x86_64

[mwilck]

Here are the dependencies in the chroot environment. I'm not sure why the debug output shows that the uas kernel module is being installed, since the usb-storage kernel module does not have a dependency on it.

We have this soft dependency on (open)SUSE:

# cat /usr/lib/modprobe.d/70-softdep-usb_storage.conf
# uas devices can be unpredictably a fallback for both drivers must be present
# (bnc#862397)
# but avoid circular dependency (bsc#1168661)
softdep usb_storage post: uas

The bug that this is supposed to solve is ancient (2014). It's well possible that this softdep isn't necessary any more. But as usual with things like that, it's hard to get rid of them, especially after such a long time, because we can't be sure whether users still rely on it.

But it doesn't cause endless recursion on plain SUSE systems. So I suppose this happens if SLE and RHEl are mixed together like in your case.
(We actually did observe recursion as the comment above says, but we solved by turning a previous hard dependency into a softdep. It's difficult to say why this doesn't work with the RHEL kernel).

I have no experience with this kind of setup. But as you're essentially building for RHEL, I suppose you should be using the soft dependencies that are valid and tested for RHEL, rather than those of the host system.

[rcorujo]

@mwilck Thank you for your response. The recursion does not occur if instead of using the dracut --sysroot <chroot-dir> option, I simply run chroot <chroot-dir> dracut ....

[mwilck]

Ok, can this be closed then?

[rcorujo]

Yes. I thought that --sysroot causes dracut to operate strictly within the chroot environment, but apparently it does not. I guess it was a misunderstanding on my part. Thank you.