Subprocess gets stuck on rsync URL check

[pitastrudl]

Yesterday it seems that all the mirror checks have failed. This might have been due to a broken rsync URL from one mirror where the rsync URL was open for connections but upon opening the connection has not sent any data back but just ACK messages (judging by tcpdump). This causes the rsync command to not timeout and hang forever.

archweb/mirrors/management/commands/mirrorcheck.py

Line 189 in 910e428

_, errdata = proc.communicate()

It was suggested in #archlinux-devops that we would need better handling of calling the subprocess as it does not timeout by python but waiting for the command to do so.

Example mirror url:

rsync --verbose  --contimeout=10 --timeout=10 --ipv4 rsync://mirror.aktkn.sg/archlinux/lastsync /tmp/tmprouz7ye4/lastsync

From tcpdump, these messages come every 2-5 seconds.

14:19:40.013414 IP mylocalip.47716 > 118.189.200.147.873: Flags [P.], seq 178:182, ack 71, win 502, options [nop,nop,TS val 880188326 ecr 4070543718], length 4
 14:19:40.296029 IP 118.189.200.147.873 > mylocalip.47716: Flags [.], ack 182, win 509, options [nop,nop,TS val 4070548723 ecr 880188326], length 0

Something like this could be implemented: https://stackoverflow.com/questions/36952245/subprocess-timeout-failure

[pitastrudl]

Either removal or restart of the servers have unstuck the checks. For now everything is back to normal.

[lcarilla]

#558 proposed a fix PR

[jelly]

This PR is now deployed live