Skip to content

Conversation

@mweibel
Copy link
Collaborator

@mweibel mweibel commented Dec 16, 2025

This fixes a couple of race conditions with node stage/unstage etc. calls.

adjust udevadm calls
udevadm when called concurrently may potentially redo symlinks.
Adding a lock, so if multiple pods attach at the same time, we don't
call udevadm concurrently.
Adjusts the order so we first check if the device is available, before
doing udevadm calls.
Adjust order of udevadm calls itself: first trigger, then settle.

Reason:

  • settle waits for new events to finish
  • trigger triggers new events but doesn't wait for them

improve node stage volume mounting

  • add volume locks to avoid concurrent operations on the same volume
  • add detection if symlinks don't add up to the right volume
  • increase logging
  • test: add concurrency/race condition tests

lint issues

  • fix ptr/nonptr receiver
  • add lint target and fix lint errors

@mweibel mweibel requested review from alakae and disperate December 16, 2025 16:43
@mweibel mweibel force-pushed the release/3.5.7-prerelease branch 6 times, most recently from 4f1856c to 39b17bf Compare December 23, 2025 13:38
udevadm when called concurrently may potentially redo symlinks.
Adding a lock, so if multiple pods attach at the same time, we don't
call udevadm concurrently.
Adjusts the order so we first check if the device is available, before
doing udevadm calls.
Adjust order of udevadm calls itself: first trigger, then settle.

Reason:
- settle waits for new events to finish
- trigger triggers new events but doesn't wait for them

Improve logging and increase the amount of logs (in debug level) to
potentially debug issues with more data.
- add volume locks to avoid concurrent operations on the same volume
- add detection if symlinks don't add up to the right volume
- increase logging once more
- test: add concurrency/race condition tests
- fix ptr/nonptr receiver
- add lint target and fix lint errors
@mweibel mweibel force-pushed the release/3.5.7-prerelease branch from 9952c20 to cbcd901 Compare January 12, 2026 07:52
@mweibel mweibel changed the title 3.5.7-prerelease fix race conditions on stage/unstage volume Jan 12, 2026
@mweibel mweibel marked this pull request as ready for review January 12, 2026 08:02
@mweibel
Copy link
Collaborator Author

mweibel commented Jan 12, 2026

workflow release charts failed due to me rebasing/squashing commits.

@mweibel mweibel force-pushed the release/3.5.7-prerelease branch 3 times, most recently from 7c89877 to e60759c Compare January 15, 2026 09:48
@disperate disperate removed their request for review January 15, 2026 10:02
@mweibel mweibel force-pushed the release/3.5.7-prerelease branch 2 times, most recently from 9d559f6 to 74e8df1 Compare January 15, 2026 12:39
adressed review comments and cleaned up CHANGELOG etc. to prepare for stable release
@mweibel mweibel force-pushed the release/3.5.7-prerelease branch from 74e8df1 to 904c815 Compare January 15, 2026 12:43
@mweibel mweibel merged commit 70ec972 into master Jan 15, 2026
1 of 2 checks passed
@mweibel mweibel deleted the release/3.5.7-prerelease branch January 15, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants