Replies: 3 comments 7 replies
-
The root of these issues is a "round peg, square hole" problem. A library/controller simply can't provide the same level of functionality for devices that report state changes on their lifeline, devices that Hail a controller (hinting that something changed, but forcing the controller to poll everything and figure out what changed), and devices that don't report state changes. If it were possible to do this, there never would've been a need for the band-aid solution the Hail Command Class provided, or Z-Wave Plus Certification going as far as mandating a Lifeline association group. In my opinion, giving users the impression that even the oldest devices can have feature-parity with newer ones is misleading. I propose doing a little test, for giggles. Do this on a "real" Z-Wave network, i.e., not with a bunch of devices lying on your desk in direct range of each other - build something that tries to resemble reality, or use the network in your home if you have one. Send a few As the library author, you understand this limitation. A user with a 3-node Z-Wave network making a feature request for polling likely doesn't. Maybe there's lots of these kinds of users. Maybe you do want the library to support their use case. At least now you know there's a very low feasibility ceiling and you can prepare yourself for the issues related to network traffic and latency this will generate. I alluded to this in one of the other discussions, but really ask yourself: how much time do you spend looking at a screen with the "status" of your lights? Sure, a lot during development and when giving demos, but in real use cases? Device and System manufacturers figured this out a long time ago. When's the last time you saw a wireless switch, that's not directly wired to the load it controls, having an LED indicating what the state of the load it controls is?
The problem is when you're polling you're not seeing the current state of your devices, are you? You're seeing the states the devices had the last time you polled them - the state that's in the cache. You'll buy into the illusion that what you're seeing is their current state if that cached information is right the majority of the time, and that depends on how often the devices' states change and what the poll interval is. Look around at the commercially-available wireless control systems that are based on polling - the one thing they have in common is that there is a single controller in the network through which all state changes happen. When you can't guarantee this, the experience falls apart.
We can discuss that, but I agree it's not the preferred approach for most. On the flip side, the question of having the network state cached locally, correctly, isn't about whether it's practical, it's about whether it's possible. You'll inevitably have the scenario where the cache is wrong. However, there's lots of successful applications and UIs that do this and live with these issues - I'm not trying to sell you on the idea that you shouldn't have a cache - I'm just pointing out that it can't be trusted, no matter how much work is thrown into it, and in my opinion shouldn't be an area of focus in the beginning.
I guess you mean toggle a switch from the UI. This must happen, or the UI feels unresponsive. When you're the one sending the command, you know exactly when and how to verify it worked, so this is entirely doable.
I would argue that because this management operation is done very rarely, it's preferable to have this operation take "a long time" and always show fresh, valid data over constantly polling this information for changes in the background, only to sit in a cache and not be able to guarantee the information is still valid when presenting it. Note that Non-Listening (NL) nodes are an interesting exception. You can't request fresh data from them when they're sleeping, but because they're required to send their Wake-Up Notification to one, and only one, controller, you can guarantee that whatever cache you maintain for these devices is valid if you're the recipient of the Wake-Up Notification. Again, I'm not trying to say that caches are evil and shouldn't be used - just that they should be used "responsibly."
What would the library use the information in these two bullets for, that is so critical to have at startup?
NIFs are required by Z-Wave to be static throughout the lifetime of a device in a network, so the supported Command Classes can't change from what was collected during the first interview. Their versions could, if a firmware update was performed, but seeing as how (with very few exceptions) Commands Classes are backwards compatible and OTA updates a rare event, refreshing the information at every boot cycle doesn't really seem to be worth it. While we're on this topic, it's worthwhile pulling up the Z-Wave Plus v2 spec. and reading up on their initial device interview recommendations. Pulling all of this information down in one go is not a good idea. I know of cases with devices based off of CR2032 batteries that can't supply constant current for too long that brown-out during these long interview processes. It's important to remember that Z-Wave is meant to be a Command & Control protocol, and that care must be taken when using it to transfer data.
As with above, static endpoints are static. Dynamic endpoints can change after the restart of the library, so that cached information can't be trusted to be valid.
I can see why this would be useful, but this doesn't sound like the library's responsibility. The library isn't going to clean-up automations defined in any of the clients, so why should it take over responsibility of cleaning up Associations? Even if the library does this, it would be irresponsible to blindly trust that what's in the cache is valid and start removing associations - it'd have to verify them anyway. The same applies to scenes, etc. |
Beta Was this translation helpful? Give feedback.
-
|
I think this is a very valuable discussion. It's clear that having a cache is not the intended way - and I'm not trying to defend the strategy of caching everything - but I'm trying to understand how to best use them responsibly while still managing to meet expectations.
Interesting experiment - will do. My network is not that large, but 15 devices should be enough. Regarding my question I was a bit unclear. The first 3 bullet points were meant for the first start/interview of a device. Point 4 for the first startup (and for the dynamic endpoints on the following startups).
Let me give you an example where I actually see no other way than to do this or to go against the S0 specs. The ID Lock keeps reusing nonces until it has requested and received a new one. Since it can only be operated securely, this bug makes it impossible to interact with unless we have identified the device and know that our S0 implementation must adapt.
It certainly feels like many people do. I did a quick search in the ioBroker forums and dug up the first visualization I found:
Do you mean displaying stale data or working without a cache? When looking at the Fibaro Homecenter UI, I'm getting the impression that they're doing the same as the mentioned visualizations. Since I don't have one, I can't verify. I'm thinking that the best middle ground in terms of staying within the bounds of what Z-Wave was designed to do would be this:
IMO this depends on where we draw the line. If you expect the driver to be an standards-compliant way to interact with the network at a medium to low level, then certainly not. If you expect the library to effectively be a black box around Z-Wave that does not require much knowledge and hides the implementation details from you, then I'd argue yes. |
Beta Was this translation helpful? Give feedback.
-
I would advise against designing the library around edge cases - design for the general scenario. In my opinion, having performance suffer for everyone just to accommodate a few outliers that are not following the rules isn't a good design compromise. I've dug into the case you linked above, and I don't think you're dealing with a device that (incorrectly) reuses NONCEs. The door lock seems to have an isolated bug in the way it's handling a secured Version Get command when there's no Lifeline configured. So how are other Z-Wave controllers capable of handling this device? My guess is they do one or both of these:
In some of the logs linked in that issue, I see the library trying incredibly hard, with lots and lots retries, to complete the interview process. Maybe this should be a topic we discuss separately?
I meant displaying state from a cache. Basically every user-facing UI will cache state, and for likely the same reasons you're looking into caching - they can't rely on reaching the driver fast enough. That compounds the problem if the library also has a cache: they're now caching the library's cache, and now there's two caches that need to be updated and kept in sync!
That sounds sensible. My recommendation would be to focus on the first bullet first, and make the second bullet largely independent of the driver - it should really just be a client that leverages the driver. |
Beta Was this translation helpful? Give feedback.
-
|
@hanskroner wouldn't it make sense to cache rarely-changed stuff, like associations and configuration parameters?
My thinking is that these values never change, unless the user is involved - and when they do change, they change via the controller (zwave-js). There should not be other controllers on the network that change stuff - there can be, but then a user is just asking for trouble :)... or at least they probably know what they're doing and will trigger some kind of refresh/update command. What I'm seeing right now, as I rebooted my controller to get the latest and greatest, was a long series of updating of configuration (at least pushed to MQTT.. I actually don't know if zwave-js issued the get's) ... which I'm 99.99% didn't change since my reboot. Will the lifeline association notify on configuration/association changes? If so - I would venture that zwave-js, as the controller node, will never actually miss a change notification.. Even if another controller were to update something, zwave-js would know about it due to the reports. And finally, I don't envision zwave-js being offline for many seconds during the day :) |
Beta Was this translation helpful? Give feedback.
-
If the devices actually send reports, see #1417 |
Beta Was this translation helpful? Give feedback.
-
It very well might, depending on your goals, but in my opinion not at the driver layer. Associations and Configuration parameters are Command Classes. While there's some complex Command Classes, like Security or Multi-Channel, that blur the line, Command Classes are the domain of the application.
In the discussion here, I was mentioning that Non-Listening (NL) devices are "special", in the sense that the Wake-Up Command Class they're controlled with was designed to allow only one device to communicate with them. A cache works great here, because the rules for invalidating it are finite and well-defined.
It will be. There's, unfortunately, always a trade-off. Then again, how often will you be configuring parameters or changing associations once you've setup your network? Maybe it's not terrible to have to wait a few seconds while the configuration/association values are loaded those few times setup is needed. In my opinion, it's not the library who is in the best position to make this decision - but the clients.
In my opinion, multiple controllers are a huge advantage Z-Wave provides. But let's table that discussion for another time, since a device doesn't have to be a controller to assign Associations or set Configuration parameters - any device that can control those Command Classes can do it. Anyone could build a little "energy monitoring box" that joins as a slave node, and then asks all What does a situation like the above mean for a generic driver like the one we're discussing here? To me, it means that if it choses to keep a cache of these things, it accepts there's no reliable way to invalidate said cache and it can't guarantee the information it holds is valid. It can try very hard to keep the data fresh, at the cost of periodically polling and ferrying lots and lots association and configuration values over the RF, but it still can't make any guarantees. The driver has no way of knowing if this is "good enough" for all its clients - maybe it's better to let each client make the choice to maintain their own cache? In each of our deployments, we could chose to setup our Z-Wave network in a way that, by convention, only one device in the network is capable of performing changes. This would make it the user's responsibility to never include a device like my "energy monitoring box" example - I can already picture the support forums full of "Device XXX broke my network" and "What's a known-good configuration for device YYY" posts... Z-Wave Certification has explicitly disallowed the building of controllers that only handle this scenario - controllers need to handle all types of networks - though requirements have been somewhat relaxed lately, so it might be allowed these days, if that's a path zwave-js would like to go down.
Somehow, the "Lifeline" name doesn't seem to do a good job at conveying its intended use :/ Not all changes are reported out the Lifeline. Even if they were, they're reported to a single device - you'd need per-command Command Class Association Groups if more devices (as in the example above) were also interested. Slave devices are very often constrained embedded systems and don't have the luxury of lots of slots for their Association Groups. |
Beta Was this translation helpful? Give feedback.
-
|
Have you been looking at the control specification which also outlines the minimum interview following an inclusion: https://www.silabs.com/documents/login/miscellaneous/SDS14223-Z-Wave-Command-Class-Control-Specification.pdf |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, that's how I structured most interview methods. But I do it on every restart, not only after the inclusion, which causes long stretches with a lot of potentially unnecessary traffic. |
Beta Was this translation helpful? Give feedback.
-
|
The value DB is persistent, isn't it? As I've noted elsewhere, I find it unlikely that zwave-js wouldn't be on the network for longer periods .. at most, I suppose you'd be out for some seconds.. As the value DB is persistent, you can immediately serve values that are just as valid as what you could poll from the devices. In the odd case that some value changes, and is announced via associations - and you miss it - I'd say bad luck.. The user is probably aware the change could be missed, as they were involved in the restart. |
Beta Was this translation helpful? Give feedback.
-
|
With #2030, we no longer query everything on restart if it was queried before. |
Beta Was this translation helpful? Give feedback.
-
@hanskroner, @LordMike: A bit of followup to #1355 - I also had it on my list of questions. This is kinda two (or three) topics in one, but they are tightly coupled IMO.
We're facing some expectations (which also touches on #1417) from users regarding what this library should do. I guess a lot comes from resistance to change...
OZW did it all a certain way (update values on startup, give feedback, etc.). This is what users are used to, this is how it has worked for the last 10 years. Some people even consider their devices broken if they don't see updates. A few examples:
I can understand these expectations - in fact I started to build zwave-js with a very similar use case in mind. ioBroker.zwave2 (what I originally wrote this for) mirrors the Z-Wave values into so-called states. Automations and visualizations use these states to display info and control devices. The entire concept builds on having the status of some system mirrored. In visualizations I want to always see the current state of my devices. For example, here are some switches displaying if a light is on or off:
I don't think it is practical for visualizations to refresh the displayed values on demand due to the inherent delay. Similarly, if I toggle a switch, I expect feedback that it worked.
The same can be said for management interfaces. I could build a UI for managing associations in a way that queries them on demand. If that UI would display all associations for all nodes, that list could take a long time to generate. If querying is done beforehand, the chance is very high that the cached info is good enough.
Now you could argue that this isn't anything that a driver should do. Basically it shouldn't care about all of this and just abstract away the heavy lifting: message serialization and parsing, managing queues and transactions, encryption, etc...
I tend to agree. But in any case we would need to continue doing the application layer work we already do - albeit in another module that you can opt-in to use. If we were to throw it all out, home automation systems would have to rework a lot and I fear many users would drop
zwave-jslike a hot potato.This leaves the question what we actually need to query on startup, a) to make the driver work and b) to keep the application layer up to date.
A few things that come to mind for a):
Beta Was this translation helpful? Give feedback.
All reactions