Python3 runtime often stalls in _finalize() or close() when garbage collecting

[denisrohr]

Hello,

we experienced a weird bug with sshfs if used as fsspec by dbt. Everything works fine until our main.py terminates or calls sys.exit(). The python runtime never finishes and we see 2 stopped python threads: the main thread (since python runtime is shutting down) and fsspecIO thread (since only daemon threads were left).

The underlying is issue is two-folded:

1. the python GC cleans up the SSHFileSystem object and calls its _finalize() method, this method eventually uses the syncwrapper, which queues it into the fsspecIO thread via the eventloop and waits for it to finish, stalling the python runtime
2. the fsspecioIO thread is daemonized, meaning it will be stopped at some point in the python runtime cleanup. I didn't check if the fsspecIO thread gets stopped before/after the GC or maybe it's even random.

We saw this issue pop-up in python3.12.12. It happens often, but not always! Sometimes everything stops fine.
Python3.10.12 seems to work always, meaning this race-condition doesn't appear to happen.

I believe this is why https://github.com/fsspec/sshfs/actions/runs/18882297037/job/53888458798 failed after stalling for 6h.

The code location:

https://github.com/fsspec/sshfs/blob/main/sshfs/spec.py#L104
Note: Our bug also happend before this PR #62 , meaning the PR #62 did'nt cause it. I believe this PR just made it more likely to happen somehow.

https://github.com/fsspec/filesystem_spec/blob/master/fsspec/asyn.py#L135
This daemon thread is supposed to process _finalize() and close() calls, which happen in the garbage collection phase after sys.exit()

Tbh I'm not sure how to fix it, maybe check inside _finalize() and close() if the fsspecIO thread is already stopped and then stop doing any syncwrapped calls?

I hope this diagram makes it clear:

[skshetry]

Opened #64, which will only close cleanly when we have an event loop running.
If no loop is running, there's no way to close the connection directly. We could attempt to release resources (sockets, etc.), but that would require accessing asyncssh’s internal APIs.

In this situation, the server will eventually close the connection after a timeout. However, if too many connections were closed in that way, that may exhaust connection on the server for short period of time. That is what was happening before, which I attempted to fix with #62.
That said, this should be rare, since by the time no loop exists, earlier filesystem instances should already have been garbage collected while the loop was "alive" (I hope).

[skshetry]

The tests are also getting stuck in my PR. And they are stuck in the middle, so I am not sure if that has to do with interpreter shutdown.

[denisrohr]

interesting, I will try to debug it on python3.11

[skshetry]

I have narrowed down the CI issue to this test:

sshfs/tests/test_sshfs.py

Lines 325 to 339 in dc17098

	def test_concurrency_for_raw_commands(fs, remote_dir):
	with fs.open(remote_dir + "/cp_data", "wb") as stream:
	stream.write(b"hello!")
	with futures.ThreadPoolExecutor() as executor:
	cp_futures = [
	executor.submit(
	fs.cp_file,
	remote_dir + "/cp_data",
	remote_dir + f"/cp_data_{index}_{secrets.token_hex(16)}",
	)
	for index in range(16)
	]
	for future in futures.as_completed(cp_futures):
	future.result()

pip install pytest-repeat
pytest tests/test_sshfs.py::test_concurrency_for_raw_commands --count 1000 -vvvs --full-trace

[skshetry]

I ended up tracking the test failure to mock-ssh-server, which is a test fixture and it's not threadsafe.

https://github.com/carletes/mock-ssh-server/blob/3c87ee8056eb80ce2efc7fc4ed1c39c53151bf64/mockssh/server.py#L42-L43

https://github.com/carletes/mock-ssh-server/blob/3c87ee8056eb80ce2efc7fc4ed1c39c53151bf64/mockssh/server.py#L80

These two run in different threads and can race to create the queue. Sometimes the former wins and overwrites the existing queue, dropping pending entries. Those commands never execute, the client never gets a reply, and sshfs hangs.