Use os.closerange instead of counting to 1 billion #425
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In
billiard.compat.close_open_fds
, after checking whether os.closerange exists, if it exists we did not use it, which makes no sense at all.Instead, the implementation called
os.close
,os.sysconf("SC_OPENMAX") - len(keep)
times.On most systems, SC_OPENMAX is something like 1024, 4096, or 65536.
In the latter case, calling it will already take 30ms on my system, almost noticable for humans.
But docker (and some systems) uses a really high value:
In other words, when running in docker we were counting to 1 billlion, at 2 million per second.
The single function call
close_open_fd([0,1,2])
takes almost 10 minutes there!The fix is trivial: When
os.closerange
is available, use the existing implementation. It's right there!An alternative approach would be to just list all open handles with
psutil.Process().open_files()
or so, and then just close those few.