Zwave database issues

[Chris Jackson]

Hi all,

With the new functionality that’s being introduced such as reading all the configuration from a device when it is first detected (i.e. when there is no XML file!) comes a small downside. If the database is not correct for a device, then initialisation can be delayed, or stall. For example, if there is a configuration item listed in the database that doesn’t exist in a device, the initialisation sequence will request this, but the device will not respond (in my opinion this is poor protocol design on behalf of zwave, but we have to work with it). The binding should work past this (eventually - hopefully!), but it’s not nice, and in the case of battery devices it can delay initialisation considerably due to the wakeup cycle…..

So, what to do - there are two options… 

1. Remove this functionality from the initialisation. It’s not something we’ve had before, so I could just remove it. However, it’s useful information to have, and it’s sometimes confusing to beginners as they don’t know why this information isn’t available, and why it sometimes disappears (which occurs when the XML is deleted). I’m not in favour of this, and in any case, the root cause of the issue is the database is wrong and the underlying problem will still occur in other circumstances….
   
2. Fix the database! If there are configurations in the database that don’t exist, then they should be removed. The issue here is that for some devices we add all the configuration for all versions, and not all versions of the devices firmware support all parameters. Fibaro devices are a perfect example of this! Here we need to split out the different versions into separate products - this has been done already for a couple of devices to avoid timeouts (well before the config parameter check was added to initialisation).

My preference is to fix the database when these problems arise. To me, this is the correct way since even if we remove this functionality from the initialisation stages, timeouts will still occur when you ask for an update of the configuration through HABmin, since it will still request parameters that don’t exist.

Fixing the database isn’t hard, but it does mean we need to be a little more careful when making changes so that we correctly handle the different versions. For some devices, this is probably well defined (Fibaro nicely version all their manuals and devices), for others, it may be more problematic. However, I suspect that devices that are likely to change are the ones that are better documented, so hopefully this isn’t an issue.

There are 2 ways of handling the versioning - firstly by using the type/id and setting up a separate product. This is how it’s currently done for the 1 or 2 devices that have been split to different versions, but it has issues as we need to identify what type/id are related to what version, and secondly, some devices don’t do this - they use the same type/id for different versions, and then increment the firmware version.

So, enough about the issue - how to solve it -:

The database will be extended to add a config file version tag as follows. 

<ConfigFile VersionMax=“1.8">fibaro/fgk101-18.xml</ConfigFile>

<ConfigFile VersionMin=“2.1” VersionMax="2.3">fibaro/fgk101-21.xml</ConfigFile>

Without the version tag(s) the database will work as it always has, so there’s no need to change everything if it’s not necessary. VersionMin and VersionMax are both optional, so we can just set a max to use a file up to a certain version. Version tags shouldn’t overlap though!

How to detect the issue…

Get a debug log after starting he binding and use the log file analyser to view the log. Filter the list to only display the node with the issue (by pressing the FILTER button). It should be plain where the issue is, and typically you would expect to see the initialisation stalled somewhere - you could expect to see something similar to the figure below…

This will show the error - now the database needs to be updated to remove this configuration parameter. Ideally to avoid breaking other versions we should try and work out what "Application Version" you have (HABmin displays this in the INFORMATION tab) and produce the correct versions for the device.

I hope this makes sense… I’m open to ideas of course, but I think getting the database right is important - this isn’t the first time this issue has arisen and since devices simply don’t respond to requested parameters they don’t support, it’s difficult to work around, and important that we don’t request such parameters since each timeout blocks the network from doing other things for up to 15 seconds!…

Chris

[Davor Fikais]

Hello Chris!

I've installed zwave binding from your repository couple of days ago (08.02.2015), and have done some reconfiguration, so I've taught it might be a good idea to delete all node xml files, so it configures them with the latest binding (before using this one, I've had zwave binding from november, if I remember well). After restarting OpenHAB, it configured only one Popp wall plug, but 4 aeon micro smart dimmers 2nd edition are not there anymore. In status field inside HABmin it shows ALIVE GET_CONFIGURATION. Is that the problem with device database and versions of devices you've mentioned?

Best regards,

Davor

On Tuesday, February 10, 2015 at 12:14:19 PM UTC+1, Chris Jackson wrote:

Hi all,

With the new functionality that’s being introduced such as reading all the configuration from a device when it is first detected (i.e. when there is no XML file!) comes a small downside. If the database is not correct for a device, then initialisation can be delayed, or stall. For example, if there is a configuration item listed in the database that doesn’t exist in a device, the initialisation sequence will request this, but the device will not respond (in my opinion this is poor protocol design on behalf of zwave, but we have to work with it). The binding should work past this (eventually - hopefully!), but it’s not nice, and in the case of battery devices it can delay initialisation considerably due to the wakeup cycle…..

So, what to do - there are two options… 

1. Remove this functionality from the initialisation. It’s not something we’ve had before, so I could just remove it. However, it’s useful information to have, and it’s sometimes confusing to beginners as they don’t know why this information isn’t available, and why it sometimes disappears (which occurs when the XML is deleted). I’m not in favour of this, and in any case, the root cause of the issue is the database is wrong and the underlying problem will still occur in other circumstances….
   
2. Fix the database! If there are configurations in the database that don’t exist, then they should be removed. The issue here is that for some devices we add all the configuration for all versions, and not all versions of the devices firmware support all parameters. Fibaro devices are a perfect example of this! Here we need to split out the different versions into separate products - this has been done already for a couple of devices to avoid timeouts (well before the config parameter check was added to initialisation).

My preference is to fix the database when these problems arise. To me, this is the correct way since even if we remove this functionality from the initialisation stages, timeouts will still occur when you ask for an update of the configuration through HABmin, since it will still request parameters that don’t exist.

Fixing the database isn’t hard, but it does mean we need to be a little more careful when making changes so that we correctly handle the different versions. For some devices, this is probably well defined (Fibaro nicely version all their manuals and devices), for others, it may be more problematic. However, I suspect that devices that are likely to change are the ones that are better documented, so hopefully this isn’t an issue.

There are 2 ways of handling the versioning - firstly by using the type/id and setting up a separate product. This is how it’s currently done for the 1 or 2 devices that have been split to different versions, but it has issues as we need to identify what type/id are related to what version, and secondly, some devices don’t do this - they use the same type/id for different versions, and then increment the firmware version.

So, enough about the issue - how to solve it -:

The database will be extended to add a config file version tag as follows. 

<ConfigFile VersionMax=“1.8">fibaro/fgk101-18.xml</ConfigFile>

<ConfigFile VersionMin=“2.1” VersionMax="2.3">fibaro/fgk101-21.xml</ConfigFile>

Without the version tag(s) the database will work as it always has, so there’s no need to change everything if it’s not necessary. VersionMin and VersionMax are both optional, so we can just set a max to use a file up to a certain version. Version tags shouldn’t overlap though!

How to detect the issue…

Get a debug log after starting he binding and use the log file analyser to view the log. Filter the list to only display the node with the issue (by pressing the FILTER button). It should be plain where the issue is, and typically you would expect to see the initialisation stalled somewhere - you could expect to see something similar to the figure below…

This will show the error - now the database needs to be updated to remove this configuration parameter. Ideally to avoid breaking other versions we should try and work out what version you have (HABmin displays this in the INFORMATION tab) and produce the correct versions for the device.

[Chris Jackson]

Yes - that’s the same issue. Try out the log analyser (try not to use too big a log - up to 20MB should be ok depending on how fast your PC is) and see if you can resolve it. IF not, send me the log and I’ll take a look - it should be relatively easy to resolve…

Chris

[Davor Fikais]

Hello Chris!

I was able to isolate the log messages for one of my problematic nodes with aeon micro dimmer. Screen shot of analysed log is in the attachment. Here is the data about device, found under Information in HABMin:

Manufacturer ID: 86

Device ID: 13

Device Type: 3

Version: 4

This information is same for all the nodes with those micro dimmers. Now, I'm not sure about procedure of updating database with this version info, so I would appreciate if you could help me with that.

Best regards,

Davor

[Chris Jackson]

Thanks.
What application version does this switch show?

[Davor Fikais]

Hello Chris!

It shows protocol version 7.0, application version 2.5

Best regards,
Davor

[Chris Jackson]

Sorry Davor,
One last question - the Aeon devices are a bit difficult - what do you think this device is? Can you point me at the manual or the link to where you bought it?

Thanks
Chris

[Davor Fikais]

Hello Chris!

Here is the link for ebay item I've bought:

http://www.ebay.com/itm/Aeon-Labs-Aeotec-Micro-Smart-Switch-DSC18103-ZWEU-868-42mhz-Z-Wave-smart-home-/151250449734?pt=UK_BOI_Electrical_Components_Supplies_ET&hash=item23373ab546

The full name is Aeotec Aeon Smart Micro Dimmer 2nd Edition

Here is a link for official Aeotec web manual for those dimmers:

http://aeotec.com/z-wave-in-wall-switches/877-micro-sei-2e-manual-instructions.html

There really is no reason for you to appologize, because you are helping me :) Thank you A lot for your help.

Best regards,
Davor

[Ben Jones]

Hey Chris,

I have run your latest binding on a fresh ZWave config (i.e. deleted all XMLs). I have a micro wall switch and two smart switches which are getting stuck. 

Attached are the info pages from Habmin and the log analysis for them. Note nodes 28 and 32 are the same type.

Not sure how to determine which config parameter is causing the problem tho?

Cheers,

Ben

[Chris Jackson]

Thanks Ben,
I can’t quite see the pics, but they should tell you where it’s getting stuck (the number on the end of the line provides the configuration parameter)… I’ll take a look at the log, but do you know if these parameters previously showed up in the parameters list? (i.e. - they would have been listed since that comes straight from the database, but did they have a value against them?). I’m just suspicious that there are quite a few seemingly incorrect parameters. I confirmed that for my devices the extraneous parameters were incorrect (from the Fibaro docs), and removing them fixed everything. The Aeon stuff is much more difficult since their versions, and documentation is not so good…

I’d like to try and find a way to step through this without getting stuck - the databases should still be correct, but getting stuck is not nice!

Cheers
Chris

[Ben Jones]

Sorry about those screenshots - was trying to reduce the size - obviously went a bit far!

For node 3 all messages are;

    Sent: CONFIGURATION_GET::2

For node 28 all messages are;

    Sent: CONFIGURATION_GET::1

For node 32 I am seeing;

    Sent: CONFIGURATION_GET::1

    Sent: CONFIGURATION_GET::2

[hl_at]

as it is not working on github - here you go.

If you Need any Special logs - please tell me what to do - thx!

[Davor Fikais]

Hello! 

This looks like the messages I'm getting upon trying to configure Aeon dimmers. Is it the same problem? Chris, is there any way to intercept those non-existing parameters (with try/except, or after some number of attempts - if they are not set, to set them to null/empty string/0/nil)?

Best regards,

Davor

[Ben Jones]

Hey Chris, 

Did you get a chance to look at my logs, is it clear which config parameters in these AeonLab devices are causing the issues? It looks like 1 & 2 only - correct?

Let me know how I can fix up the database and submit a PR for you to merge on your Habmin branch.

Cheers,

Ben

[Chris Jackson]

Hi Ben,
Sorry - I had a work do last night do didn’t get too much done by the time I got home :)

I’m just a bit surprised about this parameter - yes - it seems to be param 1, and maybe 2. I see it in another log I’ve received today as well… Everything indicates that this parameter should be ok (i.e. mainly the pepper database).

I’ll remove parameter 1 at least from the database - if I remember in your log, param 2 was right at the end and it might have just been a timeout or something. I do see param 2 responding in another log I’ve got, so I think this should be ok.

I’d be interested to see if it works - if it does, it would then be good to use the REFRESH button updates the parameter or if it stays blank…

I’ll load a new version shortly with this param removed...

Cheers
Chris

[Davor Fikais]

Hi Chris!

I've tried binding you've put online few minutes ago, but my Aeon Dimmers still don't go past GET_CONFIGURATION, and they are grayed in HABmin. I can post some log, if you need one.

Best regards, 

Davor

[Ben Jones]

No need to apologise mate - just know how many requests you get so wondered if you had missed mine. I just updated with your latest version and restart openHAB (I didn't delete my old XML node files - should I have?).

Attached is the log - those AeonLab devices still seem to be getting a bit stuck.

[Ben Jones]

I just tried a 'Refresh' of the config parameters from Habmin (for node 3). All parameters were populated apart from 2, which is what is stalling the initialisation routine as well I think. Likewise for node 28, params 1 + 2 are not populated by a refresh.

[Chris Jackson]

> I just tried a 'Refresh' of the config parameters from Habmin (for node 3). All parameters were populated apart from 2, which is what is stalling the initialisation routine as well I think. Likewise for node 28, params 1 + 2 are not populated by a refresh.

Ok - I’ll remove 2 as well… My suspicion mounts!

[Ben Jones]

Do I need to wipe my node XML files when you change the database like this?

Are you also updating for the DSC24 Smart Energy Switch (my nodes 28 + 32)?

[Chris Jackson]

No - updating the XML shouldn’t matter.  I’ve not done the DSC - will take a look at this in a minute… I’ve just loaded a new version - Davor - see if this also helps your system.

--
You received this message because you are subscribed to the Google Groups "openhab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openhab+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
Visit this group at http://groups.google.com/group/openhab.
To view this discussion on the web visit https://groups.google.com/d/msgid/openhab/19f39a73-f210-4bca-81ae-4e02e161693a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Jackson]

Removed param 1 from the DSC…

[Davor Fikais]

Hello Chris!

I've tried latest binding, but still with no success. Still getting stuck at GET_CONFIGURATION. 

Best regards,

Davor

On Tuesday, February 10, 2015 at 12:14:19 PM UTC+1, Chris Jackson wrote:

Hi all,

With the new functionality that’s being introduced such as reading all the configuration from a device when it is first detected (i.e. when there is no XML file!) comes a small downside. If the database is not correct for a device, then initialisation can be delayed, or stall. For example, if there is a configuration item listed in the database that doesn’t exist in a device, the initialisation sequence will request this, but the device will not respond (in my opinion this is poor protocol design on behalf of zwave, but we have to work with it). The binding should work past this (eventually - hopefully!), but it’s not nice, and in the case of battery devices it can delay initialisation considerably due to the wakeup cycle…..

So, what to do - there are two options… 

1. Remove this functionality from the initialisation. It’s not something we’ve had before, so I could just remove it. However, it’s useful information to have, and it’s sometimes confusing to beginners as they don’t know why this information isn’t available, and why it sometimes disappears (which occurs when the XML is deleted). I’m not in favour of this, and in any case, the root cause of the issue is the database is wrong and the underlying problem will still occur in other circumstances….
   
2. Fix the database! If there are configurations in the database that don’t exist, then they should be removed. The issue here is that for some devices we add all the configuration for all versions, and not all versions of the devices firmware support all parameters. Fibaro devices are a perfect example of this! Here we need to split out the different versions into separate products - this has been done already for a couple of devices to avoid timeouts (well before the config parameter check was added to initialisation).

My preference is to fix the database when these problems arise. To me, this is the correct way since even if we remove this functionality from the initialisation stages, timeouts will still occur when you ask for an update of the configuration through HABmin, since it will still request parameters that don’t exist.

Fixing the database isn’t hard, but it does mean we need to be a little more careful when making changes so that we correctly handle the different versions. For some devices, this is probably well defined (Fibaro nicely version all their manuals and devices), for others, it may be more problematic. However, I suspect that devices that are likely to change are the ones that are better documented, so hopefully this isn’t an issue.

There are 2 ways of handling the versioning - firstly by using the type/id and setting up a separate product. This is how it’s currently done for the 1 or 2 devices that have been split to different versions, but it has issues as we need to identify what type/id are related to what version, and secondly, some devices don’t do this - they use the same type/id for different versions, and then increment the firmware version.

So, enough about the issue - how to solve it -:

The database will be extended to add a config file version tag as follows. 

<ConfigFile VersionMax=“1.8">fibaro/fgk101-18.xml</ConfigFile>

<ConfigFile VersionMin=“2.1” VersionMax="2.3">fibaro/fgk101-21.xml</ConfigFile>

Without the version tag(s) the database will work as it always has, so there’s no need to change everything if it’s not necessary. VersionMin and VersionMax are both optional, so we can just set a max to use a file up to a certain version. Version tags shouldn’t overlap though!

How to detect the issue…

Get a debug log after starting he binding and use the log file analyser to view the log. Filter the list to only display the node with the issue (by pressing the FILTER button). It should be plain where the issue is, and typically you would expect to see the initialisation stalled somewhere - you could expect to see something similar to the figure below…

This will show the error - now the database needs to be updated to remove this configuration parameter. Ideally to avoid breaking other versions we should try and work out what "Application Version" you have (HABmin displays this in the INFORMATION tab) and produce the correct versions for the device.

[Chris Jackson]

I’ve just noticed that OZW implements a “ write_only” tag on some parameters - not all the same as we’re having trouble with though, but this might be the reason why!!!

[Ben Jones]

Sounds suspiciously like it!

[Chris Jackson]

Davor - what parameter is it getting stuck on now?

[Davor Fikais]

Hello Chris!

Sorry it took so long for the info, but I wasn't at home. Here is the screenshot of log analyzer, after running OpenHAB with latest binding. In the attachment is the part of log file itself, if it could help you in some way too. Nodes 2, 3, 4 and 5 are all Aeon dimmers, so it should be the same problem. I've filtered Node 2 for screenshot.

Best regards, 

Davor

[Chris Jackson]

Thanks - see if the latest helps?

[Davor Fikais]

Hello Chris!

I've tried latest binding, but Aeon devices are still stuck at GET_CONFIGURATION. Besides that, OpenHAB log window now shows some error messages too:

Exception in thread "Timer-12" java.lang.NullPointerException
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer.advanceNodeStage(ZWaveNodeStageAdvancer.java:738)
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer$IdleTimerTask.run(ZWaveNodeStageAdvancer.java:1051)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Exception in thread "Timer-13" java.lang.NullPointerException
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer.advanceNodeStage(ZWaveNodeStageAdvancer.java:738)
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer$IdleTimerTask.run(ZWaveNodeStageAdvancer.java:1051)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Exception in thread "Timer-14" java.lang.NullPointerException
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer.advanceNodeStage(ZWaveNodeStageAdvancer.java:738)
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer$IdleTimerTask.run(ZWaveNodeStageAdvancer.java:1051)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Exception in thread "Timer-15" java.lang.NullPointerException
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer.advanceNodeStage(ZWaveNodeStageAdvancer.java:738)
at org.openhab.binding.zwave.internal.protocol.initialization.ZWaveNodeS
tageAdvancer$IdleTimerTask.run(ZWaveNodeStageAdvancer.java:1051)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)

Best regards,
Davor

[Chris Jackson]

Thanks Davor,

I'll take a look at this first thing when I get home tonight. I keep thinking about buying myself some Aeon equipment for testing as they have so many 'interesting features', but I just can't bring myself to do so...

Chris

[Ben Jones]

I would steer clear! ;)

[Davor Fikais]

Thank you for all the effort you're putting into this and trying to resolve our issues. Especially after a work day.

Best regards,

Davor

[Chris Jackson]

Hi Davor,
There’s a new version which I hope fixes this…

Chris

[Chris Jackson]

Actually - hold off on this version... 

[Chris Jackson]

Sorry for the confusion - it is ok :)

[Davor Fikais]

Hello Chris!

This one did it! All nodes are green, and all node stages states ALIVE DONE. No error messages in OpenHAB log either. Thank you a lot for your time.

Best regards,

Davor

[Chris Jackson]

Hi Davor,
At the risk of breaking your now working system, could I ask you to give the attached version a try. This tidies up the WriteOnly parameter issue by saving the configuration into the XML file, and also properly handling WriteOnly parameters during normal operation (i.e. not just during initialisation).

If you see anything else ‘strange’ happening, please let me know - there are a few other mostly small changes in this version - mostly related to message priorities in an effort to try and make the binding more responsive where it matters (i.e. by prioritising messages to SET things higher than things like reading status, configuration, and polling). It’s a small change, and shouldn’t have a massive impact most of the time, but I just thought I’d mention it….

Thanks
Chris

(file renamed to DOC to get past virus scanners!)

[Davor Fikais]

Hello Chris!

I'm not at home at the moment, but I will test it as soon as I get home (in 2 hours or so). I'm glad I can help with this.

Best regards,
Davor

[Chris Jackson]

Hi Davor,
No problem - and absolutely no hurry - just when you get a chance is fine…

Thanks
Chris

[Davor Fikais]

Hello Chris!

I've deleted all XML nodes and installed attached binding. After starting OpenHAB, it found all Aeon Dimmers without any error reporting in OpenHAB log. It just didn't find 2 battery operated devices (Fibaro motion sensor and Aeon Door/Window sensor) yet, but I guess they'll be fine after wake-up (in status node stage it states ALIVE PING). Hope this is the info you need. If you need some more testing, I'll be glad to contribute.

Best regards,

Davor

[Chris Jackson]

> I've deleted all XML nodes and installed attached binding. After starting OpenHAB, it found all Aeon Dimmers without any error reporting in OpenHAB log. It just didn't find 2 battery operated devices (Fibaro motion sensor and Aeon Door/Window sensor) yet, but I guess they'll be fine after wake-up (in status node stage it states ALIVE PING). Hope this is the info you need. If you need some more testing, I'll be glad to contribute.

Hi Davor,
That’s great - thanks for the feedback. The battery nodes should come up in time, but of course let me know if they don’t.

Cheers
Chris

[Davor Fikais]

Hello Chris!

I found one problem, but I'm not sure if it's connected to the new z-wave binding. I've found out, if I try to turn off/on any light connected to Aeon dimmers from OpenHAB, they do not respond to commands. They do report changes in both HABdroid and WebUI, if I manually turn it off/on, but not on the commands from OpenHAB. Is there any new setting I should be aware of? I have associated Target for automatic meter report to node 1 (Z-Stick), and Notification on Status change to Hail CC sent (same as it was before). 

Best regards,

Davor

P.S.

Forgot to mention that it does report value of dimmer in OpenHAB log upon change, but it doesn't affect lights.

[Chris Jackson]

Hi Davor,
No - there shouldn’t be any change that should do this, but I just confirmed it’s also the same here so I’ll take a look at this (probably tomorrow now though).

Cheers
Chris

[Davor Fikais]

Hello Chris!

I've reverted to the last version I've had installed and it works, so, take your time. I'll gladly test new version when it's ready.

Best regards,

Davor

[Chris Jackson]

Thanks Davor - I know what happened so it’s easy to fix (tomorrow though :) )…

Cheers
Chris

[Chris Jackson]

This should fix it… I accidentally deleted a line so I’ve just put it back…

Chris

[Davor Fikais]

Hello Chris!

I didn't realize you've uploaded new binding last night. I've tried it now and everything seems to work - nodes were recreated after deleting all of them, and I can control lights with HABDroid and WebUI.

Best regards,
Davor