Supporting Columbia Weather Systems MicroServer hardware?

[Bill Burton]

Hello,

I'm helping out a small non-profit organization in Maine improve their web site which includes weather information and a webcam. They have a Columbia Weather Systems Pulsar 600 with the MicroServer computer. After spending a lot of time looking around the weewx.com site and github repo, there doesn't seem to be any existing driver nor have I been able to find anything close to what I need.

The Columbia Weather Systems MicroServer has the following API "features" based on the version 2.4.9 firmware from 1-Feb-2017:

• Current conditions available via HTTP GET requests to the MicroServer which makes XML or JSONP data available. The MicroServer can also be configured to send current conditions data to a configurable host via FTP every 15 seconds in either XML or CSV formats. In any case, all the data in a record is returned at once so there's no concept of loop or partial packets that need to be combined. Data is returned in either US or Metric units depending on how the station is configured in the Admin interface.
  
• Past data is archived for a year but not available via a documented API. It requires screen-scraping a page in the Admin interface to get the links to the archived files. These files are in CSV format with one file per day and one record per minute with data always stored in US/Imperial measurements.
  
• No documented API to get hardware info and settings. There is an XML config file that can be manually downloaded via the Admin interface. Right now, the only way it appears possible to access this file is to screen-scrape a form in the Admin interface. Otherwise more limited info is available by screen-scraping a status page.

So far, I have WeeWX 4.0b12 running with Python 3.6.9 on Ubuntu 18.04.3 LTS using the Simulator driver. My programming background is primarily back-end development using Ruby and Bash along with C# and Java and I've been learning Python 3 over the last month or two.

In searching through this group, I did find a thread from 2016 regarding the Rainwise IP-100 driver which also has an API supporting XML: https://groups.google.com/forum/#!searchin/weewx-development/columbia%7Csort:date/weewx-development/B9KMyT5DtAc/7KJK2kVFAQAJ. However, that driver only supports Python 2 so I'd rather start with something that's more recent and supports Python 3 if possible as I'm a noob when it comes to Python in general.

Can anyone suggest a recent driver or one that's a better staring point that supports XML or JSON by getting data via HTTP?

Thanks for any assistance,

-Bill

[Greg Troxel]

Not directly answering your question, but:

I would look into the MQTT extension -- which publishes json to MQTT and
doesn't do at all what you want. But it probably has some code clues.

More usefully I would look at

https://github.com/bellrichm/WeeWX-MQTTSubscribe

which does not do the web/json fetch but does do the ingest into weewx
part, and you'd replace the mqtt code with a loop to fetch every so
often from the station's API.

A decision you'll have to make is what to use as the "archive"
interval. It may be that you can do the fetch every 10s and call that
loop with every 300s being called archive. The 5 minute archive notion
is useful in terms of what's in the database.

[Vince Skahan]

On Sunday, February 16, 2020 at 1:24:23 PM UTC-8, Bill Burton wrote:

Can anyone suggest a recent driver or one that's a better staring point that supports XML or JSON by getting data via HTTP?

Take a look at the purpleair extension.  You'd have to run it through 2to3 to make it python3-compatible, but it basically does a http get of a url and gets the data into weewx pretty reliably.

I have a fork at https://github.com/vinceskahan/weewx-purpleair if you wanted to take a look.  The readme there points to the author's master copy.   Code should be pretty obvious to you. 

[Bill Burton]

Hello Greg,

Thanks for the suggestion on using the MQTTSubscribe driver. At this point, I'd rather see how far I can get writing a new driver - just trying get suggestions on the best one to use as a template.

I was going to stick with the default 5 minute archive interval as that seems very reasonable.

Thanks,

-Bill

[mwall]

bill,

i would suggest that you start with weewx3 and python2, and use the ip100 driver as a guide.

it is trivial to update the ip100 to python3/weewx4 compatibility

and by the time you are ready for that, weewx4 should be out of beta, so your upgrade path will be painless

(i would do the ip100 update right now, but i've got to take care of some installer issues for the weewx4 release)

m

[Bill Burton]

Hello Matthew,

I went ahead and started with the IP100 driver as a guide as after all the suggestions, there really wasn't a better one to start with. Before even asking, I'd looked through the source of a number of drivers but found that none of the drivers that ship with WeeWX use XML as an input as they are all much more low level.

Instead of taking the easier approach as you suggested, I dove in and went the harder route porting it to Python3/WeeWX4 at the same time as migrating the driver. At this point, I'm probably 80% done. Just discovered I misunderstood how the mapping works from the station and implemented it myself instead of using the logic in the constructor that also handles mapping from the weewx.conf file. As the input XML has over 50 values (many are aggregates), I wanted to only return the relevant ones which is less than 20 but now I'll simplify the parsing to return a dictionary that's more or less what's in the XML file.

If all goes well, I hope to be able to run stand-alone against test input files soon.

Thanks for the help,

-Bill

[Andrew Billits]

Is the Pulsar 600 just a rebranded Lufft WS-600?:

https://www.lufft.com/products/compact-weather-sensors-293/ws600-umb-smart-weather-sensor-1832/

[Bill Burton]

Hello Andrew,

On Saturday, February 22, 2020 at 5:34:41 PM UTC-5, Andrew Billits wrote:

Is the Pulsar 600 just a rebranded Lufft WS-600?:

https://www.lufft.com/products/compact-weather-sensors-293/ws600-umb-smart-weather-sensor-1832/

Yes, it appears that way at least from the basic specs such as doppler rain sensor, precipitation type detection, etc. and the unit does look identical. However, the Columbia Weather Systems version requires the use of their MicroServer computer. The weather station I have access to has this computer and is the only way I can gain access to the station data. So even if the Lufft WS-600 is the same sensor as the Pulsar 600, it doesn't imply it uses the same MicroServer computer for which I'm writing a driver. As a result, it's highly unlikely the driver I'm writing will work with a Lufft sensor.

Thank you and best regards,

-Bill

 

[Andrew Billits]

Hi Bill,

Yeah, from looking at the Columbia Weather offerings, it seems all (or at least most) of their products are just rebranded units. For example, taking a closer look I can see that *all* of their weather station hardware is from other manufactures. Lufft, Vaisala, Texas, etc.

The thing is (and my point with this)... I seriously doubt Columbia is developing custom firmware for this many devices from this many manufacturers. So the odds are high that your station is just a Lufft WS-600 and it's their server device that's been developed to talk the various protocols.

You would likely be able to connect your station to a Windows machine running the Lufft configuration software and set it to the Lufft UMB protocol (heck, it may already be configured that way). You would then have the documented lufft protocol to work with. There's already python project on GitHub for the Lufft UMB protocol.

Pardon the typos, I'm on my phone.

[Bill Burton]

Hello Andrew,

On Saturday, February 22, 2020 at 10:29:11 PM UTC-5, Andrew Billits wrote:

Hi Bill,

Yeah, from looking at the Columbia Weather offerings, it seems all (or at least most) of their products are just rebranded units. For example, taking a closer look I can see that *all* of their weather station hardware is from other manufactures. Lufft, Vaisala, Texas, etc.

Okay.

The thing is (and my point with this)... I seriously doubt Columbia is developing custom firmware for this many devices from this many manufacturers. So the odds are high that your station is just a Lufft WS-600 and it's their server device that's been developed to talk the various protocols.

Right.

You would likely be able to connect your station to a Windows machine running the Lufft configuration software and set it to the Lufft UMB protocol (heck, it may already be configured that way). You would then have the documented lufft protocol to work with. There's already python project on GitHub for the Lufft UMB protocol.

This station is a few hours drive away from me. When I am in the area from time to time, I still have to arrange to get physical access to it. As it's an operational unit, I can't go unplugging it and running experiments as that will interfere with logging current weather conditions not to mention that I have no way to plug an RS-485 interface into a computer. However, writing a driver that works at the Columbia MicroServer level is something I can do remotely and doesn't interfere with the hardware while I run tests on the driver. Such a driver will also work with many of the stations Columbia sells as they also use the MicroServer computer.