Please get rid of hard-coded timeouts

[madduck]

tl;dr: if a node takes a long time, then hard-coded timeouts take precedence over the --timeout command line option.

Details: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786997

[shallot]

It looks as if IO::Socket::INET6's Timeout setting is worthless, because if Munin can't connect to a dead node, /usr/share/perl5/Munin/Master/Node.pm will not actually emit the message

    if (! $self->{reader} ) {
            ERROR "Failed to connect to node $self->{address}:$self->{port}/tcp : $!";
            return 0;
    }

It just doesn't seem to happen.

[shallot]

In any event, a defensive approach would definitely be to clamp individual worker timeout with the global timeout. What would be the use case of allowing any worker to continue past the user-defined timeout?

[shallot]

It should also be noted that plugins/node.d/munin_stats and plugins/node.d/munin_update don't autodetect or allow update_rate changes to be configured, so their warning and critical thresholds are wrong after it's changed, as well as the graph_info text.

[shallot]

With munin-async, this gets better, because the master munin-update job seems to spend some time on the live nodes, and then about ~45 seconds into it, mass discards various dead nodes, which then doesn't interfere with a 60s update_rate

However, I'm still worried about the timings and will keep testing with more load

[shallot]

OK so after some time, I can say it's better, but not actually working. Clients that stall out ssh connections more than 60s are not actually timed out after the timeout set in the munin-update argument.

I tried adding this:

ssh_command "timeout --foreground --preserve-status --verbose --signal=TERM --kill-after=59s 55s ssh"

Unfortunately that just breaks everything immediately.

Can something be done about this? It's making the log full of FATAL messages because of overruns, when even a miniscule percentage of one's nodes are unreliable.

[shallot]

I suppose it needs to be said explicitly - Munin/Master/ProcessManager.pm containing hardcoded numbers for the 3 *timeout variables is clearly wrong given the configurability of update_rate, I don't see any reason why not to fix these to at least be derived from update_rate, if not properly configurable themselves.

This code seems to have been replaced completely in master in commit 5584ba3 in 2016 but that's still not released

It was made configurable in commit 4624c9e in 2020, probably need the Debian 11 version to test this :)