Wunderground import

[Pete Nevill]

Hi,

I have a WATSON W-8681 weather station which I have set up to send data direct to Wunderground every 5 minutes. In order to bring that data into my weewx database I use my router to redirect a copy of the wunderground url string to my server. I have written a user script to listen on a port for the wundergorund txt string. 

(This PWS is actually made by Fine Offset, just re-branded)  http://www.foshk.com/Weather_Professional/HP1003.html

The string looks like this...

ID=XXXXX&PASSWORD=XXXXX&indoortempf=78.8&tempf=58.3&dewptf=55.0&windchillf=58.3&indoorhumidity=39&humidity=89&windspeedmph=6.5&windgustmph=7.4&winddir=179&absbaromin=30.17&baromin=30.36&rainin=0.00&dailyrainin=0.02&weeklyrainin=0.04&monthlyrainin=0.39&yearlyrainin=3.56&solarradiation=259.80&UV=2


I extract each of the values and map them to the associated weewx parameters.

The trouble I am having is getting the right values for rain into weewx.

I have mapped the following:

Wunderground url                        weewx

'rainin'                  to                   'rainRate'
dailyrain               to                   'dayRain'
weeklyrain            to                   no weewx equivalent.
monthlyrainin       to                    'monthRain'
yearlyrainin          to                    'yearRain'

As far as I can tell rainin (wunderground) is current rain rate in mm/hr and daily rain is total rain since 12 mindnight.

I am guessing either my mapping is wrong or the way these values are calculated within weewx is an issue. As you can see from the URL posted above dailyrain is 0.02 but my weewx is showing zero.

I am guessing there are a few possible issues here.

1) correct mapping of parameters.
2) correct unit mapping/conversion
3) method for computation of daily
3) method for computation of rain rate
4) hardware versus software computation.

Any guidance of how to get the correct rain data into weewx would be most appreciated.

Thanks,

Pete.

 

[Andrew Milner]

You could just do it the easy way and use the weewx utility wee_import

http://weewx.com/docs/utilities.htm#wee_import_utility

[gjr80]

Hi,

To answer some of your questions. WeeWX records rain on a per interval basis, so the rain field in an archive record holds the amount of rain that fell in the archive period concerned. Day, week, month, year etc values are calculated from this underlying data. The WU dailyrainin field is the rainfall since midnight in inches. You would need to calculate the difference between consecutive records to get the rain per interval required by weeWX. The WU rainin field is the rainfall in the last hour in inches. You could use this as rainRate or you could have weeWX calculate rainRate from the rain data. There has been a lot of discussion on calculation of rainRate, have a search of the forum if you want. If you have weeWX calculate rainRate then the StdWXCalculate service (http://weewx.com/docs/usersguide.htm#StdWXCalculate) can take care of it. You can ignore weeklyrainin, monthlyrainin and yearlyrainin; weeWX does not store these, rather they are calculated as required.

How you deal with units (not just rain units) will depend on how you get your data into weeWX. Almost all WeeWX records have a field 'usUnits' that stores the unit system (US, METRIC or METRICWX - have a look at bin/weeWX/units.py) used by the observation fields in the record concerned. If you are adding data to a record then you need to observe the records unit system. If you are creating your own record for weeWX to process you can usually use whatever unit system you want as weeWX will take care of any conversions.

It is not clear how you are getting your data into weeWX. If you are inserting directly into the database you are making things tough for yourself, there are a lot of internal goings on that you will need to take care of, certainly more than just inserting data into a table. You are far better if you let weeWX take care of as much as possible. In fact, if there is no weeWX driver for your station you may be best served implementing a driver based on the fileparse driver (/home/weewx/examples/fileparse). The fileparse driver reads data from a formatted text file.

Gary

[Pete Nevill]

Gary,

Many thanks for your reply. This is very helpful.

First issue fixed. units from Wunderground url as you say are in in/hr so I fixed that, i.e.  /25.4.

I have written my own driver based on one called socketlogge from here https://github.com/poblabs/weewx-socketlogger

in my weewx.conf file i specify the driver and parameters.

[W8681]
 # This section is for the W8681 wifi series of weather stations.

  # The driver to use:
 driver = weewx.drivers.w8681

[SocketLogger]
 # This section is for the SocketLogger driver

 # Station info:
 hardware = Generic intercept for wunderground urls

 # IP address, port number and timeout in seconds of the socket to connect to
 mode = sniff # driver mode either sniff of listen
 host_ip = XX.XX.XX.XX # in sniff mode ip of weather station else should be localhost for listen
 host_iface = eth1 # only required in sniff mode
 host_port = 80 # port to listen on or port to sniff
 timeout = 320 # timeout in seconds
 # this should be greater than the interval set on your pws

 # The driver to use
 driver = user.socketlogger


I have attached the py script (still work in progress so use at own risk). Passes data to weewex with (for example) _packet['temp']. Not sure if this simply inserts data into the database or whether weewx actually "processes" the data.

I am just trying to figure out how the rainrate is handled. 
Is weewx smart enough to know if I provide a rain rate of mm/hr every 5 minutes what the cumulative rainfall is? Just started running my script with a constant rainrate for 24hrs, will see what happens. So far the rain rate on the web interface is correct. Although the "Today's Rain" still says zero. Will let it run for a few days (we should have a total of 48mm tomorrow). 


Pete.

[gjr80]

Hi Pete,

A few thoughts for you as you continue...

I know you have just posted a small extract from your weewx.conf so we cannot see your complete setup, but be careful with your 'drivers'. The weeWX architecture basically supports one 'driver' and multiple 'services'. So for example some of the folks that have a PWS and one or more one-wire sensors might be running the driver for their PWS and a one-wire service to pick up data from their (separate to the PWS) one-wire sensors. It might be just semantics but something to keep in mind. If you are just using a socketlogger.py-based driver then I would expect that woudl be your driver and thought should be all you need weeWX wise.

As mentioned previously, the weeWX rain field holds the total rainfall that ocurred over the period concerned. For loop data (ie data that comes in regualrly from your PWS, typically several times each archive period) this is the total rainfall since the last loop packet. For archive data (ie the accumulated data over an archive period, say 5 minutes) this is the total rainfall over the archive period. WeeWX can calculate rain rate from rain data but cannot calculate rain from rain rate data. If you look in the socketlogger driver you will see that each data packet it receives has a running total, the socketlogger keeps track of the previous running total value and calculates the difference (refer check_rain() method). This difference is what is yielded by the driver.

'Getting data into weeWX'. In its simplest form the driver decodes data from the data source (nominally a PWS), does any processing required (eg calculates rain from a running total) and then yields a loop packet. This packet may contain data from some or all sensors. The weeWX machinery then accumulates these loop packets, and at a fixed interval (the archive interval), runs the accumulated data through various services before it is stored in the database and reports generated based on the accumulated data. The process then repeats until the next archive interval is complete. The drivers job is to continuously decodes raw sensor data and publish loop packets. Of course there is a format for the loop packets coming out of the driver; they use a python dictionary with a number of mandatory fields (dateTime - a timestamp for the packet and usUnits - the unit system used in the packet) and one or more observation fields eg outTemp, inHumidity etc. These observation fields are typically a weeWX archive table field name though they do not have to be. Basically, if your driver emits outTemp and outTemp exists in the weeWX database schema, weeWx will automatically store the outTemp data in the outTemp related fields in the database. The key method in a driver is genLoopPackets(), this method is responsible for yielding the loop packets used by the rest of weeWX.

I am not sure if it was the process of posting to Google groups or if it was in the source file, but it appears that the socketlogger.py you posted has a tabs in it. Python has some strict indentation rules and whilst tabs can be made to work, it is likely that you will eventually trip yourself up as spaces get mixed in with the tabs. Better to get yourself a decent editor and set it to uses spaces only (when you hit the tab key).

If you have not already read them you might find the following give some insight into how weeWX works and how to customise it:

- Customization guide - Introduction
- Customization Guide - Porting to new hardware
- Notes for Developers
- various drivers, fileparse.py (examples/fileparse/bin/user/fileparse.py) is a basic one others are in /home/weewx/bin/weewx/drivers, file size is probably a good indicator of the simpler v more complex drivers.

Hope this helps.

Gary

[Pete Nevill]

Gary,

wow... now I feel stupid... The answer was staring me in the face...(even had a warning)

def check_rain(self, daily_rain_counter):
                # *** DO NOT use the &rainin= data! ***
                # Handle the rain accum by taking the &dailyrainin= reading ONLY.
                # Then submit those minor increments of daily rain to weewx.
                rain = 0.0
                current_rain = float(daily_rain_counter)
                if self.lastrain is not None:
                        if (current_rain >= self.lastrain):
                                rain = float(current_rain) - float(self.lastrain)
                self.lastrain = current_rain
                return rain

Although looking at this, is there an issue with flip after midnight? Assuming it is raining and the bucket tipped just after midnight but before the packet is sent we will have a situation where current_rain < self.lastrain (but not zero). self.lastrain will be updated but rain will be zero. Next packet will be the difference, and so miss the first bucket tip of the day. Now on a 5 minute interval that I have set up I am sure this is that significant but just thought I would raise it as a potential problem.

As such I added an else (where current_rain < self.lastrain, simply passes total since midnight):

          rain = float(current_rain)

I appreciate your comments regarding careful use of drivers etc as well as space/indents in my python script. As you can tell I am enthusiastic amateur...!

I am still a little lost as to why my daily rain value reads zero after forcing (faking) the rainrate (constant 2mm) for over 24hrs.

Thanks again,

Pete.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/4MWfKEqNO2A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pete Nevill]

SOLVED (Hopefully)....

Three steps to get this working.

1) As Gary pointed out. Use the cumulative rain (since midnight) to compute the rain since last update.So that the total rain _packet['rain'] is correct for the update period (in my case this is 5min).

2) Only send this information to weewx, ignore all the other rain parameters, i.e. let weewx handle that.

3) as per step 2) set in weewx.conf calculations section "rainRate" and "dayRain" to "software".

Initial signs are looking good. Although after 24hrs of heavy rain here in West London we now have blue sky!

Thanks for all you help and guidance Gary. Just need to extract my correct rain data from Wunderground and update weewx db...
 

Pete.
F.Y.I.
here is a link to my PWS on wundergound. I have a webcam attached, note I have overlain with imagemagik output from weewx (info created using cheetah) on the webcam image. Hope you like it.

https://www.wunderground.com/personal-weather-station/dashboard?ID=ILONDON542

[Andrew Milner]

If you are taking historic data from the WU databases why not use wee_import which does all the rain calculations from wunderground to weewx for you – together with any other conversions as required.

 

I can understand your position for live data if you are sniffing the data BEFORE it is uploaded to WU, but your last comment implies you are using historic WU data – in which case wee_import does the jobe perfectly AFIK.  I certainly used it OK for 3 yrs worth of data.

 

 

Sent from Mail for Windows 10

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

 

 

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/4MWfKEqNO2A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[gjr80]

As you can tell I am enthusiastic amateur...!

That's no problem, I don't consider my self anything more that.


On Friday, 19 May 2017 19:04:09 UTC+10, Pete Nevill wrote:

SOLVED (Hopefully)....

Good news!
 

Three steps to get this working.

1) As Gary pointed out. Use the cumulative rain (since midnight) to compute the rain since last update.So that the total rain _packet['rain'] is correct for the update period (in my case this is 5min).

2) Only send this information to weewx, ignore all the other rain parameters, i.e. let weewx handle that.

Yes, exactly right, almost another rule of thumb 'if weeWX can do it then leave it alone and let weeWX do it'
 

3) as per step 2) set in weewx.conf calculations section "rainRate" and "dayRain" to "software".

You can omit dayRain from weewx.conf, it won't cause any harm but it isn't doing anything either - rainRate is the only rain related 'observation' that weeWX can derive from other observations. You might see dayRain appear in loop packets or archive records from some stations, but weeWX does nothing with it. If you are interested you can see what weeWX can derive if you look at class WXCalculate in bin/weewx/wxservices.py.

Initial signs are looking good. Although after 24hrs of heavy rain here in West London we now have blue sky!

Thanks for all you help and guidance Gary. Just need to extract my correct rain data from Wunderground and update weewx db...

If by this you mean importing historical (as distinct from live) WU data then as Andrew said wee_import is your friend.
 

 

Pete.
F.Y.I.
here is a link to my PWS on wundergound. I have a webcam attached, note I have overlain with imagemagik output from weewx (info created using cheetah) on the webcam image. Hope you like it.

https://www.wunderground.com/personal-weather-station/dashboard?ID=ILONDON542

Looks pretty smart, am sure you are better versed in London weather than I but if I had those clouds around here I woudl say it won't be too a long for some rain.

Gary