Oregon Scientific LW301/302

[Sergei]

Hello everyone,

I have never used weewx before, but since Oregon Scientific support is non-existent and the osanywhereweather service availability is like 50% coupled with the fact that it keeps data for only 7 days, I am looking at some alternatives.
I see that not much support exists for LW30x series, but a simple packet capture reveals the following data posted by their router to their website:

66......P....x.j.P...?|..mac=0004a368e639&id=8e&rid=af&pwr=0&or=0&uvh=0&uv=125&ch=1&p=1
6.......P.x6<....P...v...mac=0004a368e639&id=90&rid=9d&pwr=0&gw=0&av=0&wd=315&wg=1.9&ws=1.1&ch=1&p=1
66..`...P....5w0>P....4..mac=0004a368e639&id=84&rid=20&pwr=0&htr=0&cz=3&oh=90&ttr=0&ot=18.9&ch=1&p=1
66......P.m23..xPP.......mac=0004a368e639&id=82&rid=1d&pwr=0&rro=0&rr=0.00&rfa=5.114&ch=1&p=1
6.......P....`..>P...o...mac=0004a368e639&id=c2&pv=0&lb=0&ac=0&reg=1803&lost=0000&baro=806&ptr=0&wfor=3&p=1
66..`...PE?l<..m.P.......mac=0004a368e639&id=90&rid=9d&pwr=0&gw=0&av=0&wd=247&wg=1.9&ws=1.1&ch=1&p=1

I'm not much of a weather person, so I don't really know what exactly all these values mean (except mac address, channel of the temp sensor), but I am pretty comfortable redirecting all the traffic destined for their service to my local web server. It is straight forward for me to capture this data then using a servlet (or equivalent) and parse it. I am just not sure what to do with it thereafter, but I would like to have a nice graphical representation of this data on either my PC or Android device.

Not sure if there's already a driver for this. I couldn't find anything on the web site.

So my question is, are there any plans to develop this driver (I'm sure there are plenty other weather stations with IP capabilities) or is there a way to get all these values into weewx easily?

Thanks,
Sergei

[mwall]

sergei,

there is not yet support for lw301/lw302, but it should be easy to implement.  there are a few approaches:

1) run a sniffer on the network, and publish the sniffed packets on a socket.  this can be done in a single command line such as 'tcpdump -x -l -tt -i vr0 port 25050 | nc -l -k 9999'.  the weewx driver then reads from the socket and parses the data.  for an example, see Erik's haculink for acurite bridge.  http://geekfun.com/hackulink/

2) run a sniffer on the network, save the data to file.  the weewx driver then reads the file.  this is the approach used by george nincehelser in ipwx for acurite bridge.  http://nincehelser.com/ipwx/

it would be nice to do this generically, since, as you note, there are now many stations that use this pattern (oregon scientific lw301/302, lacross c84612, fine offset wh2600, acurite bridge, netatmo, withings bathroom scales, some garmin devices, air mentor, foobot air quality monitor)

if you are not comfortable writing a driver, then you could help by posting the pcap data from sniffing the network traffic.  when you post pcap data, please post not only the pcap data, but also the corresponding real values and/or actions that caused the data.  for example, post pcap data and a corresponding/overlapping csv dump of data from the web site to which the lw301/302 is sending.

m

[Sergei]

Thank you for your response.
My biggest problem is the lack of time.
I have a dd-wrt based router at home which I can use to redirect the traffic from the LW301 to another server (I have a tomcat running for C-BUS services) and just dump it into a file with all the parameters it posts.
Isn't weewx using some kind of sql database to store collected data? And a web server to serve html pages?

[Sergei]

I got sometime today to have a look at the data this unit posts to the Oregon website.
I simply applied some iptables rules to my dd-wrt router to direct the traffic from the Oregon router to a tomcat instance running on another server:

iptables -A PREROUTING -t nat -p tcp -s 192.168.0.125 -j DNAT --to-destination 192.168.0.230:8080
iptables -A FORWARD -d 192.168.0.230 -p tcp --dport 8080 -j ACCEPT
iptables -A POSTROUTING -t nat -j MASQUERADE

What I found is it executes HTTP POST to /update, but no query parameters according to the server logs.
So I simply wrote a servlet and made it write the body of the request into a file.
The body of the request simply contains a string that looks like URL query parameters:

mac=0004a368e639&id=c2&pv=0&lb=0&ac=0&reg=1803&lost=0000&baro=806&ptr=0&wfor=3&p=1

I then stripped the mac address from the string and wrote data into a separate file based on the sensor id. I further split the data into the weather related info and sensor state info, which now looks like this:

[19:30:08 07/01/2016]
ttr=0
ot=28.4
Misc data: rid=9c pwr=0 ch=2 p=1

"ot" being the value of the temperature sensor which reads 28.4 degrees Celsius.
Under misc data I left the values that seem to indicate sensor properties and are not related to the weather info.

I then let the station run in this mode for a few hours and collected data into separate files for analysis.
From what I can see, the station will update sensor info every time there's a change in value or every 30 sec if there's no change.
The barometric pressure appears as a separate sensor, although it is read from the main temperature unit on channel 1.

So my setup is as follows (based on sensor ids):

8e = UV sensor
90 = Wind sensor
88 = Temp sensor (pool - ch2)
c2 = Barometer
84 = Temp sensor (outside - ch1)
82 = Rain sensor

I attach the files with collected data in case anyone wants to look at them.

What would be the best/easiest way to now get this into the weewx?
(bear in mind that I have not even had a look at weewx yet)

[mwall]

On Thursday, January 7, 2016 at 12:45:45 PM UTC-5, Sergei wrote:

What would be the best/easiest way to now get this into the weewx?
(bear in mind that I have not even had a look at weewx yet)

sergei,

take a look at the fileparse driver in the extensions directory.  that driver will read a file that contains name-value pairs.  so if you save a file that contains names that match the wview schema (the default schema used by weewx), then you can use the fileparse driver as-is.

m

[Sergei]

Hi Matt,

Yes, I have actually got sometime to install weewx and look around a bit.
I initially used the Simulator driver and added a file reading code with mappings which worked quite well.
But looking at the fileparse.py, this is a much more generic solution. I just did not realise it existed :)

So what I did was added a code to my servlet to get all the parameters I understand from the sensors and then buffer until all sensor data is updated, at which point it just writes them to a file using parameters I looked up in the sqlite db that weewx generated:

UV=4
radiation=164
windDir=22
windGust=0.9
windSpeed=1.4
inTemp=27.0
barometer=843
rain=0.00
outHumidity=39
outTemp=27.7

Then fileparse.py simply reads that file in.
So far it's been working well, but I cannot understand 2 things:

1) Rain: I see in sqlite db you have "rain" and "rainRate". My station sends 2 values: rr=0.16 and rra=36.736 for example. "rr" only goes above zero when there's rain, while "rra" will increment during rain and stay at that level. I am assuming that "rr" is the actual rate at which the level changes in the bucket (I know very little about weather). My question is whether I should be writing "rr" into "rain" or "rainRate"?

2) UV: UV index is working well, but my station is sending an additional value (UV index is sent as "uv") which is under "uvh" and this value is at 108 at night, but depending on the sun during the day it changes. I assumed it was radiation and hence wrote it out as such. Is this correct at all?

I am also reposting the data back to Oregon Scientific cloud service from the servlet, but I get more bad gateway (http 502) errors than successful posts.

Interestingly, the LW301 is just a "black box" router between USB WMR200 and ethernet, so in theory one could just use a Raspberry Pi or similar with your WMR200 driver.

Thanks for all the help...

[mwall]

On Wednesday, January 13, 2016 at 7:37:27 AM UTC-5, Sergei wrote:

1) Rain: I see in sqlite db you have "rain" and "rainRate". My station sends 2 values: rr=0.16 and rra=36.736 for example. "rr" only goes above zero when there's rain, while "rra" will increment during rain and stay at that level. I am assuming that "rr" is the actual rate at which the level changes in the bucket (I know very little about weather). My question is whether I should be writing "rr" into "rain" or "rainRate"?

sergei,

did you get this sorted?

most consumer stations use a tipping bucket to measure rainfall.  the station simply counts the number of bucket tips, so they report a monotonically increasing value for rain.  they typically have some way of resetting the counter (sometimes by power cycling the station).  multiply the number of bucket tips by some constant to get the amount of rain.

the driver should report the amount of rain in each loop or archive period.

so if the station is reporting rain total, just report the difference between each reading as 'rain'.  be sure to put it in the right units (see the tables in the docs for METRICWX, METRIC, and US).

weewx will calculate the rainRate automatically for you.  but if the hardware reports a rain rate, the driver should send that along as 'rainRate'.
 

2) UV: UV index is working well, but my station is sending an additional value (UV index is sent as "uv") which is under "uvh" and this value is at 108 at night, but depending on the sun during the day it changes. I assumed it was radiation and hence wrote it out as such. Is this correct at all?

some stations report solar radiation (typically in watts per square meter), others report a uv index (typically 0-16), and others report luminosity.  you can convert between solar radiation and luminosity, but any conversion is a pretty rough approximation.  there are too many variables involved for an exact conversion.

m

 

[Sergei]

Hi Matt,

Hmmm... Not entirely sure. I can't seem to figure out the rain units the station is using, although the actual pattern seems correct. OregonScientific cloud service seems a lot more stable these days and makes it easier to try and correlate the data.
So I decided to go with rra and just use the difference between each reading like you suggested. This seems to produce the correct graph based on the actual amount of rainfall, but on the wrong scale. (I'm using metric btw.).
I figured may be the station is reporting the rain in inches looking at the delta values produced (they're fractional with 2 decimal places). So I simply converted to mm before giving it to weewx as 'rain'.
Yesterday and last night we had some serious rain so I attach the graph from weewx and what the OregonScientific cloud server recorded for my station.
Here are the values extracted from Oregon:

All data are in 5 minute block.,,,,
utc,date,time,rainrate (mm/h),rainfall (mm)
2016-02-17T13:00:00,2016-02-17,15:00,0.0,0.0
2016-02-17T14:00:00,2016-02-17,16:00,27.8,25.4
2016-02-17T15:00:00,2016-02-17,17:00,6.5,2.1
2016-02-17T16:00:00,2016-02-17,18:00,2.8,0.0
2016-02-17T17:00:00,2016-02-17,19:00,2.2,0.0
2016-02-17T18:00:00,2016-02-17,20:00,0.0,0.0
2016-02-17T19:00:00,2016-02-17,21:00,0.0,2.0
2016-02-17T20:00:00,2016-02-17,22:00,0.0,0.0
2016-02-17T21:00:00,2016-02-17,23:00,0.0,0.0
2016-02-17T22:00:00,2016-02-18,00:00,0.0,0.0
2016-02-17T23:00:00,2016-02-18,01:00,0.0,0.0
2016-02-18T00:00:00,2016-02-18,02:00,0.0,0.0
2016-02-18T01:00:00,2016-02-18,03:00,0.0,1.1
2016-02-18T02:00:00,2016-02-18,04:00,0.0,0.0

On the graph you can see weewx recorded about 170mm between 16:00 and 17:00. The weekly stats page says the highest is 527.3 mm/hr at 16:53:43 (Wednesday) which is way off.
OregonScientific only shows total rain on its graph, but from extracted data above, the peak is 27.8mm/h with 25.4mm total for that period.
I have the original raw values for those times, but they're not of much help to me to figure out the difference in readings.
I'm pretty sure I need to stop converting the output from the station and leave as is.

Thanks,
Sergei

[mwall]

sergei,

would you be able to test a driver for the lw300 station?

https://github.com/matthewwall/weewx-interceptor

m

[Sam Roza]

Hi Matt,

I have an LW-301 here, and I have loaded the interceptor driver, but it seems that the interceptor basically requires putting a piece of hardware in between the router and the OS Internet Gateway if you don't have administration rights over your router-mine happens to be provided by my ISP, and I cannot do a DNS takeover, or put in custom firewall rules to redirect traffic to another box on the network.

I'm trying to figure out how to intercept the traffic *without* putting another piece of hardware in between the devices-for expense, power, and complexity's sake. I do not know why, but wireshark doesn't show any packets bleeding over from the gateway to my laptop via the switch they are plugged into. 

[mwall]

On Wednesday, September 28, 2016 at 5:09:13 PM UTC-4, Sam Roza wrote:

Hi Matt,

I have an LW-301 here, and I have loaded the interceptor driver, but it seems that the interceptor basically requires putting a piece of hardware in between the router and the OS Internet Gateway if you don't have administration rights over your router-mine happens to be provided by my ISP, and I cannot do a DNS takeover, or put in custom firewall rules to redirect traffic to another box on the network.

I'm trying to figure out how to intercept the traffic *without* putting another piece of hardware in between the devices-for expense, power, and complexity's sake. I do not know why, but wireshark doesn't show any packets bleeding over from the gateway to my laptop via the switch they are plugged into. 

hi sam,

you're right - somehow you need to get between the os internet gateway and the internet.

a network switch limits traffic to each physical port - your laptop will see only its own traffic, not that of other devices plugged into the switch.  a few low-end switches have an option to mirror traffic to a designated port, but that kind of feature is typical of higher-end 'managed' switches.

you could insert a network hub (not a switch) between the os internet gateway and your switch, then sniff the traffic on a computer plugged in to the hub.  but that requires one more device (the hub).  hubs are still available, but these days most people just get a switch with port mirroring if they need to tap the physical interface.

if there is wifi between the os internet gateway and your switch, then you might be able to sniff traffic there.

you could insert a low-end, low-power computer with two network interfaces between the os internet gateway and your router.  technically this would not be an additional device, since you need a computer on which to run weewx anyway.

another option is to buy a sdr dongle - about 20$US.  then use the sdr driver to sniff the rf signals directly from the os sensors.  it means another device (the dongle and a low-power computer such as a raspberry pi on which to run it), but you could rationalize it as a backup to the os internet gateway :)

m

[mwall]

On Thursday, September 29, 2016 at 22:44:13 PM UTC-4, Sam Roza wrote:

Yeah, you're right about switches, of course. Well, I guess it's time to break down and buy a Pi...

A Pi3, and some bridge work, and I should be up and running.

Matt, would you suggest the DNS methods over the other capture and scrape methods (like the tcpdump captures and whatnot)? If so, I'm wondering how to get a Pi to do DNS resolution just for this one device-though if I'm bridging the ethernet port to the wifi driver, I'm guessing that it would pass through and not be possible. So then I'd be back to the tcpdump methods, which everyone here seems to be quite familiar with (and therefore, support might be a bit easier).

sam,

the dns approach is easiest, but of course that is possible only if you control the dns on your network, or if you can configure the weather station to use a machine you control for its dns queries.  is there any way to configure the os internet gateway?  can you tell it which dns server(s) to use?  if so, then install bind on the pi and bob's your uncle.

if you have to use the pi3 to bridge, then you'll have to plug the os internet bridge into the pi ethernet port and connect the pi to your network via wifi.

alternatively, add a usb-ethernet dongle to the pi and bridge from wired to wired.

technically you do not even have to bridge - you just need to make the os internet bridge think that the pi is the os weather server.

m

[Sam Roza]

Matt,

I've got the hub in place, and the rPi set up. I have weewx installed, as well as the interceptor driver. Unfortunately, it looks like the interceptor's internal web server is causing issues with getting weewx successfully up and running.

I''m working off of your 'Example 4':

~~~

Example 4: weewx is running on host 'pi', which has a web server on port 80
to display weewx reports.  Configure the driver to listen on port 9999.  Add a
reverse proxy to the web server configuration to direct traffic on port 80 from
the device to port 9999.  Add a DNS entry so that traffic from the device is 

sent to 'pi' instead of the cloud.

~~~

Where my pi has weewx running on it, and I would like to run a web server on 80 (I've installed lighttpd). I can't figure out where to set the alternate port in the interceptor driver, though-the only place I could find where this was even mentioned was in the interceptor.py file , where I set 'DEFAULT_PORT = 9999' and then ran 'wee_config --reconfigure' again and restarted everything, but that didn't open up the socket to 9999 like I had been hoping it would.

So now that I've got the hardware portion of this all figured out and in place, I would sincerely appreciate if I could get a little bit of direction as to getting interceptor and weewx working, if you don't mind. I'm feeling a wee bit dense after grinding on this with no results this evening.

Thanks.

[Sam Roza]

Just a note for those who come after me:

There are no notes in the documentation at this moment regarding changing the interceptor default port, but to change it, you just need to add 'port = NNNN' so the driver stanzat of weewx.conf.

-Sam