Cgit is currently unavailable

[nikromen]

• copr-dist-git crashed, reboot ended up with the "press ctrl+D" prompt, mounted the data volume to the STG instance, run fsck manually, remounted back, did chmod -x on the cgit binary.

Originally posted by @FrostyX in #15

[praiskup]

FTR, here is the recent botnet attack @ Savannah:
https://lists.gnu.org/archive/html/savannah-users/2025-01/msg00000.html

[praiskup]

170212
[root@copr-dist-git httpd][PROD]# grep 14/Jan ssl_access_log-20241208  | grep cgit | wc -l
10998
[root@copr-dist-git httpd][PROD]# grep 15/Jan ssl_access_log-20241208  | grep cgit | wc -l
11387
[root@copr-dist-git httpd][PROD]# grep 16/Jan ssl_access_log-20241208  | grep cgit | wc -l
9834
[root@copr-dist-git httpd][PROD]# grep 17/Jan ssl_access_log-20250118  | grep cgit | wc -l
161555
[root@copr-dist-git httpd][PROD]# grep 18/Jan ssl_access_log-20250118  | grep cgit | wc -l
170212

2025-01-17 and 20215-01-18 shows significantly higher throughput.

This is also interesting:

$ grep 17/Jan ssl_access_log-20250118  | grep cgit | cut -d' ' -f1 | sort | uniq -c
... first ~100 users have at most 14 accesses ...
      6 77.111.111.111
      9 193.92.200.156
      9 2a00:23c6:6e8d:fb01:509a:af42:c511:3834
     11 80.109.108.129
     14 2003:cf:d729:dc99:7119:b8a1:80da:ec3b
    262 47.79.99.1
    265 47.79.121.89
    267 47.79.123.136
    269 47.82.0.13
    269 47.82.0.98
... the rest ~500 users have ~250-350 accessess ... all in 47.79.x.x - 47.82.x.x...

IOW, similar issue to Savannah

[praiskup]

See also #3595.

[praiskup]

see also: https://fosstodon.org/@[email protected]/113868168298182149

[mkurz]

Any ETA when cgit will be up again?

[FrostyX]

@mkurz I think somebody will start working on this next week, but I have no idea how hard it will be to solve this. We think we were DDoSed by AI scrapers like many other projects (please see the mastodon link above). And the last time I checked, it wasn't so clear what to do about it.

Does the disabled cgit break some of your workflows? Maybe there will be a workaround that you can use until it is fixed.

[mkurz]

Any news?

[FrostyX]

We discussed it last week and there was an idea about restoring the previous status quo and simply enabling the cgit instance again. The problem is, we expect to get DDoSed again. And once it happens, it will disable the whole machine, causing build queue to not get processed and so on. And if it happens during the night, the outage could be for many hours. So this is probably not worth the gamble.

Other idea was to limit the cgit resources through cgroups. Once DDoSed, it will probably shutdown just the cgit instance, leaving the rest of the system unimpaired

Lastly, we discussed using some kind of AI tarpit, but we don't want to maintain it. Since this is as global issue, we hoped maybe Fedora Infra team could come up with a solution for the whole Fedora ecosystem.