Unreliable behavior of munin master when there are no new spoolfetch data available

[shapirus]

In case when there are no new data available from munin-async when munin master connects to the target node and gets the due spoolfetch data, the graphs for which the new data were not produced become excluded from the content generated by munin-httpd, both html and graphs themselves.

Steps to reproduce:

1. set up munin-master with a node in an asynchronous configuration (munin-async+munin-asyncd)
2. wait until the spoolfetch dir gets populated with some data gathered from munin-node
3. remove all generated files from /var/lib/munin on the master server
4. run munin-cron
5. visit the munin-httpd page and see that all the graphs are present and displayed correctly
6. immediately run munin-cron again before the next munin-asyncd run pulls new data from munin-node
7. visit the munin-httpd page again and see that there are now no graphs whatsoever
8. run munin-cron after munin-asyncd has pulled some new data and see that the graphs reappear.

It appears that it is a political rather than technical question of what we want to do about the graphs for which there is no recent data: hide them altogether or display them with empty space for which there are no data (yet).

To me it sounds like the best solution would be to always display all graphs even when their data are stale until they reach a certain (ideally user-configurable) age of not being updated.

A curious consequence of current behavior is the race condition which I encountered during testing a single-node munin master + munin-node with munin-async/spoolfetch setup.
Currently munin-asynd launches its data gathering runs on multiples of 5 minutes: 00, 05, 10 etc. At the same time, munin-cron is by default configured with the same crontab schedule: " /5 * * * * ".
Sure thing, it so happens that munin-update connects to the target node (self in my case) and runs munin-async to fetch the spooldir data at the very moment that the same spooldir is being populated by munin-asyncd.
This results in an incomplete data set received by munin-update and consequently missing graphs. In my case it was a curious behavior like "everything is fine after the first munin-cron run, but some graphs become missing starting from the second run and never reappear".
In my setup I solved this by changing the munin-cron schedule to " 1-59/ * * * * " so that it runs one minute after munin-asyncd runs, by which time all spoolfetch data should usually be ready.

[shtrom]

This behaviour is still present on munin 2.0.49-1 on Debian and munin-* 2.0.47 on OpenBSD.

[sumpfralle]

Does anyone has an idea, where the code decides to not generate the respective graphs?

[shapirus]

Too bad I don't remember it now, I've been not using munin since the end of 2016. However the code must be somewhere in what munin-cron executes not long before graph generation. It should not be too difficult to trace it. I think I had actually found it, not even being very familiar with munin code in general.