Ability to see how long it has been since last successful hardware update

[Krellan]

Currently, there is no way, in the dbus-sensors D-Bus interface, to see how long it has been since the time of the last successful sensor update from live hardware.

If the sensor value is stable, that is, it hasn't changed in some time, there is currently no way for software to distinguish between a good sensor with an unchanging value, and a stuck or broken hardware sensor (or a buggy software driver) that is no longer providing sensor value updates in a timely manner.

As sensors are commonly used for thermal control, this is a real problem, and can lead to hardware damage if, for example, the fans are wrongly set to a slow speed, even though the system is overheating, because there was a bug somewhere in the thermal sensor and it got stuck reporting a cold temperature when in reality the temperature has increased over time and we just didn't know because the sensor got stuck.

I'm open to suggestions as to what would be a good interface addition for this feature, and to do it in a way that doesn't lead to spamming D-Bus with too much additional traffic.

[mdmillerii]

Thinking aloud, having a separate interface with the last reading time would allow it to not be transferred or subscribed, but obtainable on consumer restart for it's local policy . The consumer could then update it's local timestamp when it reads a changed value.

This also covers using a Hysteresis filter to suppress dbus update notifications.

Further including a query method "have you confirmed the reading since time X' would allow a simple boolean response (and allows virtual sensors or others republishing eg Pldm or remote satellite controller values to act on demand or publish based on subscribed events but not have a local polling configuration). Without adding additional transfers. Such a query could be used to multiple producers.

An advanced consumer could even implement I'm already elevated due to another zone in it's freshness policy.

But look likewise I believe we have a convention of responding with NaN for unreadable values, for when we are unable to access a sensor that is being polled. Unless you are concerned with publisher dbus getting stuck and the value changed notification being lost. I don't know if some publishers are trying some advanced policy like only nan after N soft failures.

None of these can distinguish a broken sensor where the sensor is not reading the expected parameter eg a broken or disconnected thermal couple.

[edtanous]

there is currently no way for software to distinguish between a good sensor with an unchanging value, and a stuck or broken hardware sensor (or a buggy software driver) that is no longer providing sensor value updates in a timely manner.

This is what the DBus functional interface is for. If a daemon detects that a sensor is not behaving, it should be setting functional to false, to flag that the sensor is in error. The correctness of a "freshness" of an arbitrary sensor is almost always a per-device decision, so it's pushed into the individual applications to decide. Some sensors might need power rails, power states, drivers, or system applications to be in the correct state for the sensor to work. Those conditions not being present very likely isn't an "error", so clients consuming the sensor interface shouldn't be relying on any sort of per sensor timing.

If we want a method call for "is this sensor healthy", we could use dbus.ping, which is what systemd is already monitoring (albeit with a much larger timeframe than most sensors would probably want). Alternatively, we could also consider enabling the systemd watchdog in our service files https://github.com/openbmc/bmcweb/blob/6136e852e4eadf9d3e83b58be17f393a96ea2f64/config/bmcweb.service.in#L13 which can make sure that the main loop is always running, and if it locks up, the service is restarted promptly.

Finally, one of the limitations we have in the sensor scanning loop is that the way the sensor dbus interface is organized, we need one message per sensor update. If we add a new property for "last updated" time, now we've halved the number of sensors DBus itself can support.

[mdmillerii]

If we add a new property for "last updated" time, now we've halved the number of sensors DBus itself can support.

Note I explicitly proposed a design that only transfers this information on suspected failure.

systemd watchdog... which can make sure that the main loop is always running

I don't think you addressed my potential scenario using a subscription to parse events from another device. While a TCP session check could confirm reachability of the remote event service I don't expect sensor failing to produce standard events. I didn't look for one and could be surprised. I doubt BMC web is actively doing anything beyond crash dump available and not said event to an unhealthy sensor even if subscribed . is there an expectation that all sensors should report failure during startup? Is this reported in health ? I didn't look. It'd be noisy.

However this is a theorhetorical argument from my side at this point and will not suggest further until a potential user arrives.