Migrate from gevent to asyncio

[rvencu]

Is your feature request related to a problem? Please describe.
Currently we see random conflicts with gevent and errors steaming from gevent on some hosts when using asyncio to parallelize multiple parallel-ssh clients

Describe the solution you'd like
We need reliable parallel ssh solution that can scale to hundreds and thousands of remote nodes. We need to create parallelism on two levels: different groups of nodes and multiple nodes on each group, so we need to introduce async programming in top of parallel-ssh to manage these groups in parallel too. Having an asyncio compatible parallel-ssh solution would be best.

Describe alternatives you've considered
tried asyncssh but the level of parallelism it supports seems to be far under what parallel-ssh can do

Additional context
congrats for the latest release, much awaited

[rvencu]

I just tested the last release and the problem is still happening.

[Mon Jan 13 11:43:30 2025] python3[216519]: segfault at 561fe6bf6390 ip 00007feb60ba3ddf sp 00007feb397f91f8 error 7 in _greenlet.cpython-310-x86_64-linux-gnu.so[7feb60b96000+10000]

[Mon Jan 13 11:43:30 2025] Code: c3 66 2e 0f 1f 84 00 00 00 00 00 48 89 c7 e9 48 26 ff ff 0f 1f 84 00 00 00 00 00 48 8b 07 48 85 c0 74 0d 48 c7 07 00 00 00 00 <48> 83 28 01 74 0b c3 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 08 48

[pkittenis]

Hi there,

Thanks for the interest.

Asyncio and gevent are not compatible. Asyncio cannot use native C libraries so is not an option for parallel-ssh, one of the reasons gevent is used.

Do you have some code to reproduce an issue you are having? Parallel-ssh has been tested to very high scales, 1000 hosts, without segfaults.

If I understand correctly you want two groups of parallel connections going at same time with different sizes and priorities. That can be done by gevent itself, it can run multiple thread loops each in its own thread.

Parallel-ssh can run on top of that thread. parallel-ssh's tunneling does this, the tunnel connection must be in its own thread so it's always sending data back through the tunnel. It would mean some changes so that the parallel clients can get a hub passed in, where gevent runs its event loop. Currently they always use the main thread's hub, other than the tunnel server.