Advice on a new driver and how to make it available

[vk3...@gmail.com]

I have written a new driver for my XC0422 weather station (which I believe is a WS1001 and HP1000 clone). It talks directly to the console via the WiFi connection using TCP sockets with no packet sniffing or other interception techniques. I'm looking for some advice on some final 'finishing touches' and how to make it available to others if they are interested.

I have been running it on my Weewx Raspberry Pi (along with a Nginx web server) for some months now and the driver code would seem to be reliable.

The console continues to send its packets to WU and the additional connection by my driver does not (seem to) interfere with this, nor the other way around.

As I said, it makes a direct connection to the console so I've found that it helps to be on the same subnet to avoid any router blocking of the broadcast messages it uses to initially find the console. After that it makes a direct connection to the console and exchanges TCP packets with it.

The Console allows you to select various units for wind speed (including Beaufort!), temperature, pressure, rain and solar radiation and the driver will read the settings from the console each time when it makes connection. If a connection is lost, then the driver restarts broadcasting again and trying to reconnect.

Because I live in a metricated country, the packet that is created is in METRICWX units. Many of the existing drivers seem to have this hard coded in the 'genLoopPackets' function but is there a better way to pick this up? If not then I believe that my code picks up the settings from the console for all values with differing units and other variables (such as humidity and wind direction) are pretty much standard across the various measurement systems.

At present the code includes 'testing code' that I've used to confirm that the various unit transformations all work, and this is controlled by a single class boolean variable. Is it better to leave this code in there for others to check things out or should it be removed to make the code easier to understand?

Finally, how is the best way to let others know of this driver so they can try it out for themselves? I'd love for others to try it out on branded WS1001 and HP1000 WiFi consoles to make sure that it works as I think it will.

Thanks

Susan

[mwall]

susan,

just to be clear, your driver works with the wifi console, not the bridge, right?

be sure to package your driver as a weewx extension.  that will make it much easier for people to install.  see the customization guide for details about how to package an extension, and see the fileparse extension as an example.

you might want to distribute it using github.  take a look at the drivers listed on the weewx wiki for examples:

https://github.com/weewx/weewx/wiki

be sure to take a look at the wiki page about the observer hardware:

https://github.com/weewx/weewx/wiki/observer

there have been quite a few different approaches and implementations for the ws1001 (and variants) hardware.  the first weewx implementations were done about 2 years ago, but they were scraping and capture/redirect approaches.  there are also at least 2 direct implementations mentioned on wxforum (one implemented in perl, another in C but not yet published).  weather-display is supposedly getting direct support for the hp1000 any day now, but it is not clear whether it uses the scraping approach or the direct query approach.

it does not matter whether you use METRICWX, METRIC, or US.  just be sure that the units are consistent for whichever unit system you choose.  also beware of the hardware - can users break your driver by changing the units on the hardware?  usually the answer is no, but you should test for this.

be sure you pick an appropriate driver name.  i think you should call it HP1000, since WS1001 is an Ambient name, and Observer and ObserverIP are brand names of Ambient Weather and are loosely applied to console or bridge.

you should also provide a 'model' attribute, so that users can specify whether they have a ws1001 or hp1000-wifi or xc0422 or OBSERVER-IP-WIFI or something else.

i'm pretty sure the driver for hp1000 can work right out-of-the-box.  that means it is a good candidate to be included in the weewx distribution once you get sufficient testing on it.

please let us know when you get it working so i can delete my observer repository.  i've had some great help from dr_bob from wxforum, but it has languished since i do not have the hardware to test it directly.

m

[mwall]

On Wednesday, January 11, 2017 at 10:01:58 PM UTC-5, vk3...@gmail.com wrote:

At present the code includes 'testing code' that I've used to confirm that the various unit transformations all work, and this is controlled by a single class boolean variable. Is it better to leave this code in there for others to check things out or should it be removed to make the code easier to understand?

imho, you should keep the testing code in the driver.  it will help you tremendously when someone starts using the driver in ways or environments you never expected.

i've been using a "run the driver directly" pattern for this kind of testing.  at the bottom of the driver file, make a __main__ section so you can invoke the driver directly like this:

PYTHONPATH=/home/weewx/bin python /home/weewx/bin/user/hp1000.py --help

it should have a minimal number of options.  if nothing is specified, it should get data from the station and print it to stdout.  there might be options for port or interface - whatever someone would have to put in the driver's stanza in weewx.conf.  then you might have an option to test decoding of data or sending a broadcast message - just the basic things that you had to do to make sure you could talk to the hardware.

if you put the test code there, then it will be runnable for debugging and diagnostics, but it will help keep weewx wee since the code in __main__ never loads when weewx actually runs.

once again, take a look at some of the drivers i've written for examples:

https://github.com/matthewwall

and if you have your own clever tips and tricks, please post so we can all learn!

m

[dr__bob]

Actually, this driver sounds very interesting.  mwall is currently developing the same kind of driver that you are probably using, that communicates with the ws1001 console with tcp (using the EasyWeatherIP protocol).  Since he doesn't have a ws1001, I'm helping him with the debugging.  But if you have a driver already up and running, that would make thing much easier!  So I can only second his suggestion to post yours on github.  

In principle, the choice of units could be made dependent on the console, as it would appear that part of the EasyWeatherIP protocol includes a "SETUP" record, which contains the units that the console is using.

At the moment, the driver that mwall has put together is still in the very very early stages, as we haven't been able to get the console to respond to the UDP broadcast packet.

Bob

[vk3...@gmail.com]

(Busy weekend - I hope to get a 'first cut' version out shortly)

I'm using the SETUP packet to get the units. At the moment I only read that packet when the driver start (i.e. if you change anything on the console you would need to 'bounce' Weewx to pick them up) but in principle there is nothing to stop you making this query each time around the loop or every so often.

The 'choice of units' I was talking about was more the 'usunits' item in the packet.

Susan

[vk3...@gmail.com]

I have attached the tar.gz file that I *think* will work as an extension but I've only managed to test it on my iMac with Weewx installed via the 'setup.py' method.

Can some mind soul(s) please try this out (in a test environment in case it generates black holes etc.) before I put it into the repository?

Also any feedback on the usefulnes (or otherwise) of the 'readme.txt' or mistakes in the 'install.py' are welcomed.

Thanks

Susan

[dr__bob]

Hi Susan,  I replied to you privately, but I should probably do it here as well so that others (more wise and experienced than I) can chime in...  I didn't use your install.py, since you just sent me the xc0422.py, but I did exactly what the installer would do:  I put it in bin/user/xc0422.py.  I'm using deb, so it went in /usr/share/weewx/user/xc0422.py.  There doesn't seem to be any defaults defined for xc0422.py.  In my weewx.conf, I had to specify all options or weewx couldn't load the driver.  I'm not sure whether that's normal behavior or not.  Here is my snippet from weewx.conf; I knew that I had to add driver and ip_address_mask, since my local net uses that instead of 10.1.1.  But I'm guessing that there must be a way to define defaults within the driver itself.

[XC0422]

    driver = user.xc0422

    ip_address_mask = "192.168.1.255"

 

    retry_count = 5

    

    socket_timeout = 5

I tried at first to just use the first two lines, but after the driver didn't load, I added the second two lines.  Now it's fine.  

One thing that I have seen is that the genLoopPackets loop just screams through.  The console seems to return data whenever it is asked.  The way the loop is written now, it asks as soon as it parses the last reply, which can be many times a second, it seems.  I was wondering whether a sleep for 5? seconds at the end of the loop would make sense.  If nothing else, it would cut down on network traffic...

The following is not really related to this driver per-se, but it is an issue which I've had to handle... 

At the moment, my home wifi network is a bit flakey (I've got to talk with ATT Uverse!).  While random non-responses from the ws1001 console are handled quite nicely in the driver, the loss of wifi is not, and weewx will exit.  To take care of that, I added a systemd configuration for weewx.  My weewx.conf looks like:

# systemd configuration for weewx

[Unit]

Description=weewx weather system

Requires=syslog.service

Requires=ntp.service

After=syslog.service

After=ntp.service

#RequiresMountsFor=/home

[Service]

ExecStart=/usr/bin/weewxd --daemon --pidfile=/var/run/weewx.pid /etc/weewx/weewx.conf

ExecReload=/bin/kill -HUP $MAINPID

Restart=always

RestartSec=60

Type=simple

PIDFile=/var/run/weewx.pid

#User=weewx

#Group=weewx

[Install]

WantedBy=multi-user.target

The "Restart=always" guarantees that weewx will get restarted, and the "RestartSec=60" guarantees that my wifi will have had a chance to reset before going into a start-crash-start-crash endless loop.

So far, the above has been running for a couple of days without issue.  THANKS Susan (and the developers of weewx!).

dr__bob

[gjr80]

Hi Susan,

No comments on the code as such but just a thought on the readme, I have put together a few extension now and the format I try to follow for readme.txt is:

• introductory paragraphs. Naming the extension, describing what it does, what it consists of and maybe a little of how it does it if applicable

• heading 'Pre-requisites'. List any pre-requisites, minimum weewx version is a good example

• heading 'Installation'

• sub-heading 'Installation using the wee_extension utility' then list the step by step instructions for installing using wee_extension
• sub-heading 'Manual installation' then list the step by step instructions for installing manually
• heading 'Uninstallation'
• sub-heading 'Uninstallation using the wee_extension utility' then list the step by step instructions for uninstalling using wee_extension
• sub-heading 'Manual uninstallation' then list the step by step instructions for uninstalling manually

I have found that when I put the extension on GitHub readme.tx then becomes a fairly good template for the readme.md that can be displayed on your extensions GitHub page. Obviously though you can make a markdown file look a bit better than a text file.

Hope this is of some benefit.

Gary

[mwall]

susan,

in addition to gary's suggestions, i would add the following:

- gary's outline for the readme contents is excellent.  please browse the other drivers from the weewx wiki for examples.

- do not bother with 'manual' instructions in the readme.  you should put detailed instructions about configuration options in the driver itself, typically in a big block of comments at the beginning of the file and docstrings for each class/method as appropriate.  then possibly use a subset of that in the readme to explain the options that absolutely must be set.

- use one of the following: readme, readme.txt, or readme.md.  readme.md is nice if you distribute from github, because it will be automatically formatted, yet it is still quite readable as plain text.  if you're doing any complicated formatting in the readme, you are probably doing it wrong.

- whichever readme you decide to use, keep each line less than 80 columns of characters.  this may sound like an arcane or anachronistic suggestion, but there are still many cases where someone must view the readme in a window/screen that is limited width.  random wrapping makes that cumbersome.

- you might want to use 'HP1000Driver' instead of 'xc0422' as the driver class name.

- set default values for all of the options that are optional.  for example, instead of this:

        self.retryCount = float(stn_dict['retry_count'])

do this:

        self.retry_count = int(stn_dict.get('retry_count', 3)

- use snake_case for variables instead of camelCase

- run your python code through pylink or pyflakes.  or just open it in pycharm and address all of the warnings.

- for required parameters that have no sane default, instead of this:

        self.ipAddressMask = stn_dict['ip_address_mask']

do this:

        try:
            self.ip_address_mask = stn_dict['ip_address_mask']
        except KeyError, e:
            raise Exception("required parameter %s has not been specified")

- in the default stanza, include only the parameters that most people will have to set.  in this case you can leave out retry_count and socket_timeout (assuming that you set defaults in the __init__ method).  document what they do in the docstring for __init__, but no need to put every parameter in the conf.

m

[mwall]

perhaps most important items as you make your work available to others:

- add a copyright line to the top of readme, install.py, and the driver .py itself:

Copyright 2017 Susan <x...@yyy.zzz>

email address is optional.

- if you want to distribute under a license, then include a file called 'license' with that license.  otherwise do 'all rights reserved' in the copyright line.

- two common open source licenses are the MIT license and the GPL license.

m

[vk3...@gmail.com]

Thanks for all of the feedback - brilliant!

Mike - I was a bit wary of claiming too much 'copyright' as I must admit I've "borrowed" quite a bit from your "pmon" extension.

I noticed that using the "wee_extension" install method only put into the config file the lines in 'config' section of the 'install.py' file. However the driver itself has the 'confeditor_loader' function that I must have got be run sometime in the past (just putting the code in the directory and using the 'wee_config' reconfigure to switch to the new driver perhaps???) and that has commented lines which I think is much better.

I'll also change the name. the XC0422 was the only clone that I had access to and I didn't want to make claims that I could not back up by calling it anything else. However it would seem that it does work on other 'clones' and so using the original name woudl be netter.

Dr Bob (name taken from the old Muppet sketch per chance???), I can put in a delay (with the delay length driven by an optional parameter with a default of 5 seconds). I must admit that the network traffic didn't bother me on my home local network (especially when you look at the garbage my Apple TV puts onto the network) but this is a good idea.

Also I'll add in the suggestion for the systemctl file. In the 'readme' I mention that you should put in references to make sure the network is up and running so the Weewx daemon will not cause problems on startup.

Gary - Thanks for the suggestions. I'll read to read up on 'markdown' and give that a go.

Back with another go shortly. Also thinking about using the 'HISTORY' data to see if there are gaps in the Weewx databse that can be filled in by the console, but that will be V2.

Any thoughts about sampling the SDETUP packet more often to detect console unit changes or is just telling people to restart Weewx OK?

Susan

[dr__bob]

On Thursday, January 19, 2017 at 8:54:26 PM UTC-8, vk3...@gmail.com wrote:

Dr Bob (name taken from the old Muppet sketch per chance???),

Indeed!  I loved that show! :-)

 

I can put in a delay (with the delay length driven by an optional parameter with a default of 5 seconds). I must admit that the network traffic didn't bother me on my home local network (especially when you look at the garbage my Apple TV puts onto the network) but this is a good idea.

Great!  

Also I'll add in the suggestion for the systemctl file. In the 'readme' I mention that you should put in references to make sure the network is up and running so the Weewx daemon will not cause problems on startup.

Well, all I did was take the util/systemd/weewx.service that you can find in the "setup.py" version of weewx and modify it for the "deb" locations and added the Restart and RestartSec commands.

Any thoughts about sampling the SDETUP packet more often to detect console unit changes or is just telling people to restart Weewx OK?

Well, if you want to worry about that, then you probably have to check the SETUP in each loop packet. Which is kind of yucky.  The really yucky part is that the output of NOWRECORD depends on what units the console 

is displaying.  

Bob

[mwall]

On Thursday, January 19, 2017 at 11:54:26 PM UTC-5, vk3...@gmail.com wrote:

Thanks for all of the feedback - brilliant!

Mike - I was a bit wary of claiming too much 'copyright' as I must admit I've "borrowed" quite a bit from your "pmon" extension.

I noticed that using the "wee_extension" install method only put into the config file the lines in 'config' section of the 'install.py' file. However the driver itself has the 'confeditor_loader' function that I must have got be run sometime in the past (just putting the code in the directory and using the 'wee_config' reconfigure to switch to the new driver perhaps???) and that has commented lines which I think is much better.

as you noticed, wee_extension will insert the config parameters specified in install.py

but wee_config will insert the (commented) parameters from the confeditor (if there is one).  this also lets you prompt the user for any required options for which there are no defaults.  you can figure out sane defaults for everything for this driver.  for the network mask, you can default to a generic broadcast.  then users can specify something more specific if they have a special network configuration (e.g., to span a vpn or router).
 

I'll also change the name. the XC0422 was the only clone that I had access to and I didn't want to make claims that I could not back up by calling it anything else. However it would seem that it does work on other 'clones' and so using the original name woudl be netter.

Dr Bob (name taken from the old Muppet sketch per chance???), I can put in a delay (with the delay length driven by an optional parameter with a default of 5 seconds). I must admit that the network traffic didn't bother me on my home local network (especially when you look at the garbage my Apple TV puts onto the network) but this is a good idea.

the typical approach is to define a poll_interval, with a default value set to the shortest sensor update interval.  then just time.sleep(self.poll_interval) in the while loop in genLoopPackets
 

Also I'll add in the suggestion for the systemctl file. In the 'readme' I mention that you should put in references to make sure the network is up and running so the Weewx daemon will not cause problems on startup.

it would be better if you do not.  instead of applying that band-aid, make the driver more robust.  you should retry the network connections, and if the retries fail more than max_tries times, then you raise a weewx.WeeWxIOError exception.  then weewx handles it as it does for any other driver.

remember that not everyone uses systemd (*bsd, windows and macos do not even have systemd)
 

Gary - Thanks for the suggestions. I'll read to read up on 'markdown' and give that a go.

Back with another go shortly. Also thinking about using the 'HISTORY' data to see if there are gaps in the Weewx databse that can be filled in by the console, but that will be V2.

Any thoughts about sampling the SDETUP packet more often to detect console unit changes or is just telling people to restart Weewx OK?

Susan

there is no reason to restart weewx.  the station is already sending you the unit system in each packet.  so just make the driver deal with the changes on the fly.  you can either change usUnits, or do conversions on each observation (you'll probably want to do the latter).

you should use snake_case for every function too (except for the ones you *must* do camelCase, such as genLoopPackets).

at some point you should refactor so that all of the test code goes into the __main__ section.  isolating the test code will become especially important when you try to do additional features such as history data.

m

[mwall]

here is a bit more unsolicited advice:

- split the code into two classes:  HP1000Driver and HP1000Station.  the station class should do all the work - that is where the network stuff happens, and that is where the decoding happens.  then you can derive a TESTStation from HP1000Station to put all the test code in.  then the driver class has an instance of the station, calls methods on it, and wraps any failures into WeeWxIOError, RetriesExceeded and other weewx exceptions.

- definitely wait to do the history stuff.

- eventually (after the history stuff?) you should be able to implement the configuration options too.  that means that you would be able to configure the hardware using the wee_device utility.  and that means you could manage the station without having to use the windows software at all.

- once fully tested, this would be an excellent addition to the list of drivers in weewx core.  it will be able to work right out of the box with no futzing, and it is becoming a pretty popular station.

m

[mwall]

finally, and most importantly,...

very nicely done!

[dr__bob]

On Thursday, January 19, 2017 at 9:19:44 PM UTC-8, mwall wrote:

 

Also I'll add in the suggestion for the systemctl file. In the 'readme' I mention that you should put in references to make sure the network is up and running so the Weewx daemon will not cause problems on startup.

it would be better if you do not.  instead of applying that band-aid, make the driver more robust.  you should retry the network connections, and if the retries fail more than max_tries times, then you raise a weewx.WeeWxIOError exception.  then weewx handles it as it does for any other driver.

remember that not everyone uses systemd (*bsd, windows and macos do not even have systemd)

Here's the error message that I get when my wifi goes down:

Jan 18 20:07:14 rpi weewx[26548]: engine: Caught unrecoverable exception in engine:

Jan 18 20:07:14 rpi weewx[26548]:     ****  [Errno 101] Network is unreachable

Jan 18 20:07:14 rpi weewx[26548]:     ****  Traceback (most recent call last):

Jan 18 20:07:14 rpi weewx[26548]:     ****    File "/usr/share/weewx/weewx/engine.py", line 847, in main

Jan 18 20:07:14 rpi weewx[26548]:     ****      engine.run()

Jan 18 20:07:14 rpi weewx[26548]:     ****    File "/usr/share/weewx/weewx/engine.py", line 186, in run

Jan 18 20:07:14 rpi weewx[26548]:     ****      for packet in self.console.genLoopPackets():

Jan 18 20:07:14 rpi weewx[26548]:     ****    File "/usr/share/weewx/user/xc0422.py", line 128, in genLoopPackets

Jan 18 20:07:14 rpi weewx[26548]:     ****      (self.ipAddressMask, UDP_BROADCAST_PORT))

Jan 18 20:07:14 rpi weewx[26548]:     ****  error: [Errno 101] Network is unreachable

Jan 18 20:07:14 rpi weewx[26548]:     ****  Exiting.

Jan 18 20:08:15 rpi weewx[26838]: engine: Initializing weewx version 3.6.2

Clearly the sock.sendto failed here, since the wifi went down and there was no network at that point.  weewx then crashes, and a minute later systemd started it up again. (And it probably got to that point because the recv further down the while loop failed.) So a try, except bracketing the sendto would help.  With a configurable delay?  With a configurable number of retries?  But I've really got to figure out why my cable modem apparently periodically restarts.  

[mwall]

On Friday, January 20, 2017 at 12:40:51 AM UTC-5, dr__bob wrote:

So a try, except bracketing the sendto would help.  With a configurable delay?  With a configurable number of retries?  But I've really got to figure out why my cable modem apparently periodically restarts.  

weewx is designed to handle these things.

the driver should retry n times, where n is an optional parameter in the driver, e.g., max_tries=3.  there should also be a retry_wait, which defines how long to wait, in seconds, between retries.

if the driver has max_retry failures, then it raises a weewx.RetriesExceeded exception.

at that point, the weewx engine does one of these things:

1) if weewx is just starting up, weewx will exit

2) if weewx is just starting up, but the loop_on_init parameter is true, then weewx will sleep for 60 seconds, then try the whole thing again.  it will do this forever.

3) if weewx has been running when this happens, weewx will sleep for 60 seconds, then start over

this way you can disconnect the network, then reconnect, and weewx will recover automagically.

so all the driver has to do is retry, and there is no need to add extra layers of systemd or cron or other types of nannies

that way you can focus on hardware features such as history!

m

[vk3...@gmail.com]

Thanks for the comments and suggestions.

I read the comments on Friday (my time) and then spent today (Sunday) making some revisions:

- changed the name to HP1000

- added the loop delay parameter (default of 15 seconds) and associated code

- added the default/error checking around reading in the parameters

- re-wrote the readme.md file with the markdown format

- downloaded pycharm and looked at the comments it made. I've cleaned up some of the points but I've also not yet taken up a number of the 'style' comments (also the comments about using snake_case and the like) - they will probably take a bit longer

- recreated the .tar.gz file (attached) and made sure that I removed the ".DS_Store" file that MacOS puts in - I noticed that having these seems to upset the "wee_extension --list" command

Of course I then realised that you have made a number of other suggestions. I'll look at those shortly but, to be honest, at this stage I'm more concerned about making sure that the functionality works. Style is great (and I'm learning Python so I'm probably carrying over a whole lot of bad habits form other programming languages) but I can clean them up a bit later on.

The network loss handling will definitely be the next thing on the list.

Susan

[dr__bob]

Hi Susan,

I tried it out.  All works fine, except, you have "driver = weewx.user.HP1000DRIVER" in the weewx.conf in your install.py.  It should be "driver = user.HP1000".  Otherwise, it seems to be working, on my Saturday evening.  

By the way, there are still some Mac-isms in your tar file, such as HP1000/._install.py.  They didn't seem to bother anyone.

Cheers,

Bob

[vk3...@gmail.com]

The latest version is attached. I *think* this is nearly read to go.

I know that there are some things outstanding (such as the coding style) but I think I have the functionality sorted, including the network failure restarts but I'd appreciate it if others can test this out - I've tried various failure modes and it seems to work but there will always be test cases I've not tried.

Cheers

Susan

[vk3...@gmail.com]

Just noticed that the beta of V3.7.0 is available.

In the release notes from Tom he noted that any missing 'windGustDir' is to be determined by Weewx with this version.

Please note that the last version of my driver (posted above) does assign 'windDir' to 'windGustDir'.

Therefore, is it better for me to test the version (below v3.7 does the copy, v3.7 and above does not)  or to take out that bot of code and just release my driver for 3.7 and above?

I guess what I'm really asking is how soon will v3.7 be released relative to my driver (which, by the lack of any negative (or positive!) feedback could be anytime now)?

Susan

[dr__bob]

Hi Susan,

I can't answer the questions about windDir and co, and I have to admit that I haven't had a chance yet to try out your latest, but I did take a look at it.  There were two things that I noticed.  The first is that you still have "driver = weewx.user.HP1000DRIVER" in the weewx.conf in your install.py.  I think that it should be "driver = user.HP1000", since your file is HP1000.py.  Also, I see that the version number is still 0.1. Bumping the version number would help distinguish the different versions.

Otherwise, your version from Jan 21 has been running perfectly in my setup since then. 

[vk3...@gmail.com]

Thanks for the feedback. I'll plead 'guilty' to not bumping the version number, but I've just downloaded and expanded the file I attached above and the line in the 'install.py' file that I think drives all this is:

                    'retry_wait' : 5,

                    'driver' : 'user.HP1000Driver'}},

Am I not looking in the right place?

Susan

[mwall]

On Monday, February 13, 2017 at 9:28:05 PM UTC-5, susan wrote:

Thanks for the feedback. I'll plead 'guilty' to not bumping the version number, but I've just downloaded and expanded the file I attached above and the line in the 'install.py' file that I think drives all this is:

                    'retry_wait' : 5,

                    'driver' : 'user.HP1000Driver'}},

Am I not looking in the right place?

susan,

1) you should rename the file to hp1000.py

2) the 'driver' field refers to the python module in which the driver code resides.  so you want 'user.hp1000' (assuming that you lowercase the filename)

3) do not include extra stuff in install.py.  all you need is this:

# installer for HP1000DRIVER                                                   
# Copyright 2017 Susan Mackay                                                  

from setup import ExtensionInstaller

def loader():
    return HP1000Installer()

class HP1000Installer(ExtensionInstaller):
    def __init__(self):
        super(ProcessHP1000Installer, self).__init__(
            version="0.1",
            name='hp1000',
            description='Driver for the HP1000 weather stations.',
            author="Susan Mackay",
            author_email="vk3...@gmail.com",
            files=[('bin/user', ['bin/user/hp1000.py'])]
            )
 
in particular, there is no need for all of the configuration options.  first of all, the driver should default to sane values when none are specified.  secondly, for drivers, the ConfEditor does the actual insertion into the weewx config file.

4) you should automatically detect the netmask so that the user does not have to specify it.  default to using the string '<broadcast>', but if a user specifies a netmask then use the netmask they specify.

5) what is the difference between 'retry_count' and 'max_retry'?  you might want to use names that are more descriptive for those parameters.

6) the hardware_name property should be the hardware model name, such as WS1001 or XC0422.  typically this is an optional 'model' parameter in the driver stanza.  is the value you decode from data[0:8] a model number, or some kind of station name set by a user?

7) you should use snake_case_variable_names instead of camelCaseVariableNames

m

[dr__bob]

Hi Susan,

this is a very minor bug, but I found it while looking through my log files.  There's a missing "%" on an strftime string.  Yup, I did say minor! Here's the diff to HP1000.py:

*** HP1000.py 2017-02-05 02:15:01.000000000 -0800

--- /usr/share/weewx/user/HP1000.py 2017-02-20 16:52:10.466575082 -0800

***************

*** 237,241 ****

  

                      loginf('Established contact at %s' %

!                            datetime.datetime.now().strftime('%d/%m/%y H:%M:%S'))

  

                      # Now read the SETUP packet to find the current value units

--- 237,241 ----

  

                      loginf('Established contact at %s' %

!                            datetime.datetime.now().strftime('%d/%m/%y %H:%M:%S'))

  

                      # Now read the SETUP packet to find the current value units

Bob

PS: the subject of this thread should probably be changed to something more meaningful/useful.

[vk3...@gmail.com]

Yep - Spotted that myself on Sunday. Thanks.

Also took the advice about removing the extraneous lines form the install.py file - much better now. (I copied this from one of the example files pretty much without understanding the relation ship between these lines and the code in the main file to add in the config options. I think I have this now but I didn't see anything about this in the docs. Perhaps this could be made a bit clearer??)

I think I'm nearly ready to release this but I'd still like a comment on hwo to best handle the new version (V3.7.0)

Susan

[mwall]

On Monday, February 20, 2017 at 9:45:55 PM UTC-5, vk3...@gmail.com wrote:

I think I'm nearly ready to release this but I'd still like a comment on hwo to best handle the new version (V3.7.0)

susan,

you should not have to do anything special for weewx 3.7.0 - there are no changes in 3.7 that would break any existing drivers.

m

[mwall]

weewx 3.7 will probably not be released before mid-march.

your driver should not emit any data other than what it gets from the hardware.

so if someone runs it with 3.6.x or earlier, they will not get windGustDir, but if they run it with 3.7 or later, they will.

m

[vk3...@gmail.com]

Thanks for the advice - I had added the WindGustDir code in (all of 1 line!) but it is trivial to take it out again.

Susan

[Robert Clare]

Hi Susan, could you please bump the version number consistently? You've got 0.2 in install.py and 0.1 in HP1000.py. And could you bump it to 0.3? I used 0.2 internally for my own station for your last version, just so that I knew what version I was using... At the moment it could easily be just you and me using your code, based on the traffic on the google user group.

Also, I think that line 556 of HP1000.py should be "driver = user.HP1000", although other more knowledgable people should chime in.

Thanks very much for all of your effort in this! The system has been running pretty much flawlessly for me since I installed your very first version. I have had some hiccups with the WS1001 console hanging, which requires a power-cycle. These *might* be related to interference between the EasyWeatherIP protocol and the operation of the console, because I didn't have as many when the console was just talking to WU. But they haven't been very common since, either.

Again, thanks very much!

Bob

> On Mar 11, 2017, at 0:20 , vk3...@gmail.com wrote:
>
> Last (I hope) test version before release.
>
> Cleaned up the code, the install process and the variable names etc.
>
> Susan
> <hp1000.tar.gz>

[vk3...@gmail.com]

Horrendous goof on my part - I hope I have deleted that last post. Somehow a syntax error crept in but this one should be OK.

I've also updated the version number to 0.3 in both the source file and the instal file.

Bob - I don't think that the line you mention needs to be changed. As far as I now it is the name of the class (prefixed by the 'user.' folder which becomes part of the nested class when it is imported) which does have the 'Driver' text at the end of it. It has been that way for a while and has always worked for me.

Thanks for checking up on me and keeping me honest.

Cheers

Susan

[dr__bob]

On Sunday, March 12, 2017 at 6:11:49 PM UTC-7, vk3...@gmail.com wrote:

Bob - I don't think that the line you mention needs to be changed. As far as I now it is the name of the class (prefixed by the 'user.' folder which becomes part of the nested class when it is imported) which does have the 'Driver' text at the end of it. It has been that way for a while and has always worked for me.

Hi Susan,  that code goes into weewx.conf.  When I first installed your driver, back before you renamed it hp1000, the code was "driver = user.xc0422Driver" and weewx complained that it couldn't find xc0422Driver.py. So I removed the "Driver" part. Since then, I've had the appropriate text (driver=user.hp1000) already in weewx,conf, and the install script doesn't touch the text already there. A new user would probably get bitten. But we should defer to the "big boys" on this. I note that mwall, one of the gurus, stated back on Feb 13:

2) the 'driver' field refers to the python module in which the driver code resides.  so you want 'user.hp1000' (assuming that you lowercase the filename)

Bob 

[vk3...@gmail.com]

Hopefully the last request: I have what I believe should be the V1.0 gz file.

What do I do with it now to make it available?

I've not done this sort of thing before and (at least in this aspect) I'm a total novice.

Thanks

Susan

[Thomas Keffer]

You need to store it somewhere, or host the repository on GitHub. Then make a wiki entry that points to its location. There are extensions that use both approaches on the wiki.

For example, here's one that's hosted externally, and here's one hosted on GitHub.

-tk

[vk3...@gmail.com]

Thanks. First time using github (as a contributor) and first time writing to a wiki (yep - Luddite First Class: that's me!)

I hope others find this useful.

Susan

[vk3...@gmail.com]

I've just updated the HP1000/WS1001/other-clones driver in the GIT repository.

If you have the 'netifaces' Python package installed then the driver will work out the network broadcast mask for itself. If the package is not installed then it will work as before. 

The actual process is:

- if the 'ip_network_mask' value is defined in the .conf file then that will be used

- if that is not defined BUT the 'netifaces' package is available, then it will look for the broadcast mask of the 'default' interface (typically the one with the gateway to the internet defined on it)

- if it still can't find anything then it will report an error

I've done it this way so that you can over-ride the automatic broadcast mask selection if it gets it wrong (for whatever reason). However what it does mean is that for anyone upgrading the driver, they will need to comment out the line in the .conf file for the automatic determination to occur.

If any brave souls would like to try it out then I'd appreciate the feedback.

Susan

[vk3...@gmail.com]

I've further updated my driver (in the github repository) to read historical records from the weather station when weeWx starts up. It uses te 'genStartupRecords' function to do this.

If the weeWx database is empty (the timestamp passed to the driver is None) then it will try to read all of the historical records from the weather station. Otherwise it will find the (roughly) corresponding history data record and read form there on.

I've also added comments in the driver to document the packets that are used.

I'd appreciate any feedback on this.

Thanks

Susan

[Francesco Scaramella]

Hi Susan!! I saw that it was available on GitHub a new version for the driver of the stations HP1000 for WEEWX, I installed it and I saw considerable changes compared to the previous version (in particular I noticed that it goes to recover all the data not yet downloaded from Station in case of locking or shutting down the WEEWX server), but I do not know why, after finishing the discharge of data from the station HP1000 (in blocks of 100) The driver seems to cease to function and consequently weewx... I'm wrong what I have there is an error in the New driver??? Thanks for this very useful driver!!

[vk3...@gmail.com]

Hello Francesco,

I'm sorry to hear that you are having a problem.

Can you let me know the error messages that you are getting - they should be in the SYSLOG?

Susan

[Francesco Scaramella]

Certainly!! Tonight just I get back from work unloading the log message of the station. To give you an indication, by calling the command sudo tail-f/var/log/syslog You can read that weewx must recover 100 packets of data from the station.... it executes.... and after downloading the 100 blocks of data weewx remains locked. Trying to give the start and stop commands starts the data discharge cycle, until the tail runs out.... part weewx, but after a while it stops processing processes again. However as soon as possible I will send you the log of the station. Beautiful The fact that the new driver retrieves all the data not present in the DB of weewx, thanks!!!

 

[Francesco Scaramella]

Here's the log of my station. The whole first part is with the old driver for HP100 WiFi. The last part is what happens in weewx after installing the new version of the driver... hopefully it is indicative!

[vk3...@gmail.com]

Hello Francesco,

The log has identified a bug which has now been fixed and a new file uploaded to GitHub. Basically I had refactored some of the network access code (to avoid duplicating the code for both the historic and current record access) and I had not transferred over a variable initialisation. The problem only shows up when there are a lot of network access failures during the retrieval of historic records.

A couple of (possibly unrelated) comments if I may:

- I suspect that your network is a little shaky in that you seem to be getting a lot of FTP errors and also my driver gets a lot of timeouts. 

- I note that the system time is updated (from NTP) some time after weewx is up and running. Weewx starts at 22:05:25, the system continues until 22:25:20 (with a big jump form 22:05:58 to 22:17:01) and then at 22:29:06, the next message is that the time has been corrected to 22:43:19. This may have an impact on the timestamps of the records written to the database. Also see the weewx Users Guide seciton on 'Clock skew' (although that talks about problems with the clock in the future rather than in the past)

- the 'action 17' error message is a well known problem with Raspbian (https://blog.dantup.com/2016/04/removing-rsyslog-spam-on-raspberry-pi-raspbian-jessie/) which can be easily stopped.

- if you are running from an SD card, then you might like to write the log files to a ram-disk. I know that this means you lose the logs after a system crash/reboot but it can preserve the life of the SD card. (I also have the whole weewx folder, including the generated HTML files, on a NFS drive on my NAS for the same reason).

Susan

[Francesco Scaramella]

Thanks Susan for the prompt correction of the driver and for the indication received. In fact I had already known the problems of My network. Unfortunately the station is installed in a country location not served by ADSL services and we share between neighbors a same connection to be able to have a little all... just keep it stable is a big job..... anyway I'll treasure the indication to bring In RAM the log!! I forgot to do it!! Thanks immensely for the help received!

[Mike Frison]

Does anyone know whether this can be adapted to the WS-2902/WH2900? 

It's a similar Fine Offset machine with wireless to console and WiFi to router connections and easy uploads to WU, ambientweather.net, and weathercloud.

I have mine sending directly to WU with iptables on the router teeing them to a Raspberry Pi 2B where the interceptor driver is feeding them to weewx, but the HP1000 driver would be a much cleaner approach.

I've tried it, but don't get a response from the UDP broadcast, just repeated timeouts. (no firewall on the pi) 

Suggestions?

[vk3...@gmail.com]

The protocol used to talk to the console is based on what I could see being used by the EasyWeatherIP app (the version I looked at was running on my iMac under MacOS).

If there is similar software that can talk to your console (or even a later version of the software I used) then there is a chance to understand the protocol. It is not easy to know how much needs to change without knowing this.

Susan

[Mike Frison]

I've tried EasyWeather 3.1 Beta on a Windows PC and it can't find the 2902 so I just sent email to Ambient support asking for protocol info.

I'll let you know what they say.

Thanks,

Mike

[Mike Frison]

Ed at Ambient says the ws-2902 doesn't have a communications protocol. ;^)

[mwall]

On Monday, September 11, 2017 at 3:50:30 PM UTC-4, Mike Frison wrote:

Ed at Ambient says the ws-2902 doesn't have a communications protocol. ;^)

you could hit it with a port scanner to see if it is listening for any network requests

then hit it with telnet on any open ports to see it whether it responds

m

[Mike Frison]

I've attached a zenmap log of a port scan.  The only port it found open was 45000 with service asmp. The suspected OS (99% accuracy) is NodeMCU firmware. For what it's worth, the hostname is ESP_0BD7E0.

I attempted to telnet to the open port

telnet 192.168.1.16 45000
Trying 192.168.1.16...
Connected to 192.168.1.16.
Escape character is '^]'.

My networking skills are negligible so I have no idea what to do next.

Mike