unlocking LUKS device using fido2 token fails roughly every other time between reboot/replugging #852
Comments
|
Oh i forgot to mention, in another system, Debian bookworm, it works every time. every single time.
|
|
One of the outcomes of the linked systemd issue was that they connected libfido2's debug logging to systemd's logging facilities. Could you please enable systemd's debug logging level and post the output of the failure case here? |
|
hi, thanks for you reply. As i noted, at first i thought it was systemd related, but now i've come to realize it's not, i tried making a crontab @reboot entry that unlocks using normal cryptsetup command, and it happens there too. So would you know how i could run a normal cryptsetup open command with libfido2 debugging enabled? I don't know if it's even libfido2 related, i just don't really know where to start looking.. I tried a few libfido commands like FIDO_DEBUG=1 fido2-token -L, but what shold i do to get debugging output when i run the cryptsetup command? I have to connect some USB cables and do USB passthrough etc to get the test VM setup again.. (again, this happens on baremetal system too, it's not related to VM or USB things) |
|
to reiterate, only now after realizing this happens when doing a manual cryptsetup open command as well, it's only now i've seen that "Token returned error during pre-flight: Input/output error" message, after hitting CTRL-C. searching for that message led me to the linked systemd issue, where libfido2 is mentioned, and that's why i'm here. So i'd appreciate any help on how to get libfido2 debuggin enabled, to maybe get some more output.. |
I'm no expert on systemd components, so take the following with a grain of salt: I'm quite sure that you're still using
You will have to set the systemd log level to "debug". IIRC, one way of doing it is setting It might be easier to call
You might still want to consult the systemd folks. Out of curiosity, does ever hang in a similar fashion? |
nope, doesn't hang.. but i've just spent several hours trying, and now suddenly i am unable to recreate the hang situation at all. Where ctrl-C is the only way out, and is then followed by the "Token returned error during pre-flight: Input/output error" message i have tried with both cryptsetup open, and systemd-cryptsetup attach / detach Right now, the hangup occurs every other boot. Because the hangup is indefinite, it won't timeout ever during boot, it's highly likely that it's that specific hangout that occurs. but again, no matter what i do, i'm suddenly unable to recreate it manually, like i was a few days ago. Occasionally i do get a: error, but that is resolved by trying again, there's no indefinite hangup. I've tried several reboots without the token plugged, so it reverts to asking for passphrase at boot, and then trying to recreate the error after boot by plugging in the token, and manually doing cryptsetup open or systemd-cryptsetup attach, but it just always works now suddenly. I can't understand it. That would again point to systemd, but again, i must be well over 100 reboots now, i've tried alot of systemd things, nothing helps. Everything points to it hanging when trying to query the fido2 token for a response forever. These last reboots it's been exactly 50/50, it hangs, then ctrl-alt-del, and next time it works. Right now i'm trying to figure out how to downgrade libfido2 in fedora 41, hopefully to the same version i have in the debian bookworm system, where it always works, every time. If i'm successful in that, and nothing changes, then it would point to libfido2 NOT being the issue. Honestly, this might be a firmware bug in the onlykey duo device. But still, it works every time in debian.. If the libfido2 downgrade doesn't change anything, and if i can't ever again recreate this manually, i'm gonna have to give up. It's not the biggest issue, since i just use it to unlock /home automatically at boot, and i don't reboot that often. When it happens, ctrl-alt-del, and next time it works. But it's still annoying as hell, just today i learnt how to enroll the fido2 token with --fido2-with-user-presence=no, and that would allow me to boot without needing to touch the key. But with this bug, i'll have no joy of that either.. if you know the command to downgrade libfido2 to version 1.12.0 using dnf5 in fedora 41, help would be appreciated, but i'm gonna try googling for that now, i'm not sure how much more time i want to spend on this today.. |
|
well i tried downloading libfido2 1.12.0 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2087781 but it won't install it using dnf downgrade /path/to/.rpm or dnf install.. it says it needs libcbor.so.0.7()(64bit) Anyways, i doubt this would help.. now suddenly being unable to recreate this error manually, i just have to give up again.. |
|
ok, another little update.. i removed everything from crypttab and fstab, and instead created a cron @reboot entry, where i manually do systemd-cryptsetup attach. First boot it worked, second boot it hangs. And when it is in that hanged state, it's then when i try to do cryptsetup open, which also hangs (because apparently there's another process stuck doing that), and then do the ctrl-C, that i get the: note, it also hangs with systemd-cryptsetup attach, but does not give the pre-flight error after ctrl-c. So.. it's not systemd, as this happens after systemd, triggered by cron @reboot entry. When i execute in this hanged state, i only get these two lines as output, as opposed to a long list of info when it's not in a hanged state: and also here i need to do ctrl-c, to get the prompt back. |
|
Since version 1.8.0, libfido2 uses locks to serialize access to the underlying When it hangs, can you see anything holding a lock on |
right, been so occupied with trying to recreate the error, forgot all about your reply.. i will try that next.. |
|
Please see my previous reply: #852 (comment) |
nothing. Now after a reboot into a state where it locks, this time it did NOT return to ask passphrase as a fallback, like it did previously in the output i posted a moment ago. also now it started working after first hanging, not falling back to passphrase, and then ctrl-c.. lemme do a few more reboots, see if i can get lslocks -u to show anything of interest.. |
|
ok first boot, now it works fine. here debug output when it works as it should after reboot: |
|
ok, next reboot, and it hangs: output from systemd-cryptsetup: This time there is no fallback to passphrase prompt after 20 seconds. so yes, now the lock is there. |
In this case, we are sending a request to your authenticator ( Whether this is due to the firmware implementation of your authenticator, your Linux kernel, libfido2, or something else entirely is unfortunately very difficult to tell. Do you happen to have another brand of authenticator available to test with? |
|
unfortunately i don't have anything else to test with. Yeah i would say firmware is a pretty likely scenario. This token does have open source firmware, available on github: https://github.com/trustcrypto/OnlyKey-Firmware Or it's a kernel bug, which would be nice, because an update could possibly fix it.. The fact it works every time in the other debian bookworm system gives me hope.. I know it's very hard to debug, because only i have the physical token, and it basically requires lots and lots and lots of reboots. Anyways, the last two outputs are from a reboot after which it works fine, and the next reboot where it hangs. That's progress.. If you come up with any ideas, let me know.. as you can't replicate this in anyway on your side, it's too difficult to debug, so don't worry about it.. Thanks for everything so far, i've learned alot |
|
Just as a note, since it works everytime in a bookworm system using 1.12.0-2, i do plan to install a VM running bookworm with the same kernel and libfido2 versions as the aforementioned, and then try to individually change kernel or upgrade libfido from backports or something, see if i can isolate what is causing this. |
|
OK, i've installed a debian test system, kernel 6.1.0-31-amd64, libfido2 (1.12.0-2) It works every time without issues. I enabled backports, but there is no newer version of libfido2 there. @LDVG , do you have any info on how to install version 1.15 in debian? would you be able to create a .deb package? Edit, i'm looking at 3 ubuntu versions now, yammy, noble and oracular. |
|
Well, installed yammy (libfido2 version 1.10), works fine, added PPA, upgrade to 1.15, still works fine. So all this for nothing, it's not libfido2, it's not systemd, it's something in fedora. Still no closer to a resolution :( Edit: i asked in fedora forums, maybe someone there could help: |
For future reference, I'd personally probably try installing from source if you want to switch between different versions.
Not at all, happy to try to help. If you get any closer to figuring it out, please let us know.
🤞 |
Still i have to ask, do you have any physical HMAC challenge-response capable physical token? All you have to do is install fedora 41 VM. Add a 1GB block device. cryptsetup luksformat it, open it and create FS on it, and then systemd-cryptenroll your fido2 token into a second keyslot. Then add entries for that to crypttab:
then add fstab entry, for ex:
then
etc etc OR just reboot the fedora VM and try to cryptsetup open or systemd-cryptsetup attach.. |
Yes, I have multiple such YubiKeys available. 🙂
Sure. Here's what I ended up doing, after installing a fresh Fedora 41 VM (QEMU/KVM): I then rebooted my virtual machine and manually tried unlocking the encrypted volume: and again and again I then realised I had a different kernel than you appear to have so I installed the available updates and rebooted. which still is not the same as yours. Nevertheless, I tried opening the encrypted volume again: I tried this multiple times again with reboots in between without issue. Finally, I tried adding an entry to My YubiKey then starts blinking during the boot process. I touch it and I am prompted with the usual display manager logon screen. After logging in, I can see the unlocked volume mounted: This behavior stays consistent across multiple VM reboots. I have not been able to get it to fail. Sorry. |
Thanks so much for the effort. I've had the issue across multiple kernels in f40 and 41. What a pain. It looks like it's the combination of fedora kernel and my model of token. Still, knowing it works with your yubikey is another valuable piece of the puzzle. Thanks again. |
What version of libfido2 are you using?
#dnf info libfido2
Updating and loading repositories:
Repositories loaded.
Installed packages
Name : libfido2
Epoch : 0
Version : 1.15.0
Release : 2.fc41
Architecture : x86_64
Installed size : 238.2 KiB
Source : libfido2-1.15.0-2.fc41.src.rpm
From repository : anaconda
What operating system are you running?
OS: Fedora 41
Kernel: x86_64 Linux 6.12.11-200.fc41.x86_64
What application are you using in conjunction with libfido2?
/usr/sbin/cryptsetup
/usr/bin/systemd-cryptsetup
How does the problem manifest itself?
Hi. I'm opening a LUKS2 device using Onlykey Duo FIDO2 device. The problem is, between reboots, it works roughly every other time. Meaning after a reboot, about 50% of the time it will work right away. And keeps working no matter how many time i do cryptsetup open/close.
But roughly the other 50% of the time, or every other reboot, it doesn't work. I say roughly, because sometimes it might not work 2 reboots in a row, and sometimes it might work two reboots in a row. But mostly it's every other reboot.
When it doesn't work, unplugging the fido token, and replugging it, giving the pin, it will then work. And it will continue to work no matter how many times i do cryptsetup open/close after that.
Same thing happens with reboot, without unplugging the device. When it doesn't work, after the next reboot it probably will, and then it will also continue to work no matter how many times i do cryptsetup open/close.
When it works it looks like this:
Asking FIDO2 token for authentication. 👆 Please confirm presence on security token to unlock.When it doesn't work it will just sit there in an endless timeout, no output on screen or in logs, when it should start blinking to confirm user presence.
Then pressing ctrl-C says this:
^CFailed to open FIDO2 device /dev/hidraw1: FIDO_ERR_RX Token returned error during pre-flight: Input/output error*** some background info: ***
I'm using this to unlock a LUKS encrypted zvol used as a keystore at boot. At the beginning, now several hundred reboot attempts ago i was unlucky in that manually unlocking happened to always work, but when systemd was supposed to unlock during boot, roughly every other time it failed. Because it had worked manually, i was convinced it was a systemd issue, and i have spend a very long time trying to debug systemd, until finally i tried manually again, and it was only then i discovered that it also happens when manually unlocking, after a reboot.
It was only then i saw the "Token returned error during pre-flight: Input/output error" message.
Searching for that message lead me to this thread:
systemd/systemd#27947
So i don't know if this is a libfido2 issue, or a kernel issue as described in that linked thread. I just thought i would start here in libfido2 github.
Please excuse me if i'm writing this in the wrong place.
I'm currently testing this in a VM, which i can easily reboot over and over, and also snapshot and rollback. I'm not a coder, and i don't know how to do 'git pulls' or systrace or anything like that, but if instructed, i can install anything in this VM and do whatever i can to help solve this issue.
I have also tested this on baremetal, and with a virtual hdd/physical ssd instead of zvol and many other things, it makes no difference.
Because a reboot almost always makes it work the next time, i'm leaning towards it being a kernel issue maybe, rather than libfido2, BUT then again, unplugging and replugging the device results in same behaviour as reboot, it will then likely work, so.. it could also be a combination of libfido2 and kernel?
After now approaching probably a hundred reboots, i would be very very glad for any help.
Regards, Mike
Is the problem reproducible?
yes
What are the steps that lead to the problem?
reboot
Does the problem happen with different authenticators?
Please include the output of
fido2-token -L.fido2-token -LPlease include the output of
fido2-token -I.fido2-token -IPlease include the output of
FIDO_DEBUG=1.FIDO_DEBUG=1The text was updated successfully, but these errors were encountered: