Support for UK Met Office WOW

[Alan Major]

Tom

Do you have any plans to add support for submitting observations to the UK Met Office Weather Observations Website? Not sure how many users you have got this side of the pond!

They give guidance here but I am not sure how I would go about set that up it in weewx. Is it something that could be easily done? Has anyone done it so far?

Alan Major
Birdham, UK

[Thomas Keffer]

Hi, Alan

No plans for me to support it, but looking through their protocol, it seems pretty simple and very similar to the existing 'Ambient' protocol, which is supported by weewx. The module you would need to extend, 'restful', was recently redesigned to make it easier for users to supply their own RESTful services such as WOW. 

Basically, you would write your own driver by defining a class, similar to class Ambient found in restful.py. You then instruct weewx to load it and post new data using it by adding its driver to a new subsection in [StdRESTful] in weewx.conf.

If you've ever programmed in Python I don't think you will find it very hard. If not, well, it's a good exercise for the student. :-)

-tk

Alan Major
Birdham, UK

--
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Alan Major]

Tom

Once again, thanks for the pointers. I managed to complete the exercise and observations are now being reported to the UK Met Office Weather Observations Web site here.

This is the code I used for the driver should it be of use to any one else who wants to report to the WOW site:

#===============================================================================
#                             class WOW
#===============================================================================

class WOW(REST):

    """Upload using the WOW protocol.
   
    For details of the WOW upload protocol,
    see http://wow.metoffice.gov.uk/support?category=dataformats#dataFileUpload
   
    For details on how urllib2 works, see "urllib2 - The Missing Manual"
    at http://www.voidspace.org.uk/python/articles/urllib2.shtml
    """

    # Types and formats of the data to be published:
    _formats = {'dateTime'    : 'dateutc=%s',
                'windDir'     : 'winddir=%.0f',
                'windSpeed'   : 'windspeedmph=%.0f',
                'windGustDir' : 'windgustdir=%.0f',
                'windGust'    : 'windgustmph=%.0f',
                'outHumidity' : 'humidity=%.0f',
                'dewpoint'    : 'dewptf=%.1f',
                'outTemp'     : 'tempf=%.1f',
                'rain'        : 'rainin=%.2f',
                'dailyrain'   : 'dailyrainin=%.2f',
                'barometer'   : 'baromin=%.1f'}


    def __init__(self, site, **kwargs):
        """Initialize for a given upload site.
       
        site: The upload site ('WOW')
       
        station: The name of the station (e.g., "KORHOODR3")
        as a string [Required]
       
        password: Password for the station [Required]
       
        http_prefix: The URL of the upload point [Optional. If not
        given a prefix will be chosen on the basis of the upload site.
       
        max_tries: Max # of tries before giving up [Optional. Default
        is 3]"""
       
        self.site        = site
        self.station     = kwargs['station']
        self.password    = kwargs['password']
        self.http_prefix = kwargs.get('http_prefix', site_url[site])
        self.max_tries   = int(kwargs.get('max_tries', 3))

    def postData(self, archive, time_ts):
        """Post using the WOW HTTP protocol

        archive: An instance of weewx.archive.Archive
       
        time_ts: The record desired as a unix epoch time."""
       
        _url = self.getURL(archive, time_ts)

        # Retry up to max_tries times:
        for _count in range(self.max_tries):
            # Now use an HTTP GET to post the data. Wrap in a try block
            # in case there's a network problem.
            try:
                _response = urllib2.urlopen(_url)
            except (urllib2.URLError, socket.error, httplib.BadStatusLine), e:
                # Unsuccessful. Log it and go around again for another try
                syslog.syslog(syslog.LOG_ERR, "restful: Failed attempt #%d to upload to %s" % (_count+1, self.site))
                syslog.syslog(syslog.LOG_ERR, "   ****  Reason: %s" % (e,))
                syslog.syslog(syslog.LOG_ERR, "url used: %s" % (_url))
            else:
                # No exception thrown, but we're still not done.
                # We have to also check for a bad station ID or password.
                # It will have the error encoded in the return message:
                for line in _response:
                    # WOW signals success with '200'
                    if not line.startswith('200'):
                        # Bad login. No reason to retry. Log it and raise an exception.
                        syslog.syslog(syslog.LOG_ERR, "restful: %s returns %s. Aborting." % (self.site, line))
                        raise FailedPost, line
                # Does not seem to be an error. We're done.
                return
        else:
            # This is executed only if the loop terminates normally, meaning
            # the upload failed max_tries times. Log it.
            syslog.syslog(syslog.LOG_ERR, "restful: Failed to upload to %s" % self.site)
            raise IOError, "Failed upload to site %s after %d tries" % (self.site, self.max_tries)

    def getURL(self, archive, time_ts):

        """Return an URL for posting using the WOW protocol.
       
        archive: An instance of weewx.archive.Archive
       
        time_ts: The record desired as a unix epoch time.
        """
   
        record = self.extractRecordFrom(archive, time_ts)
       
        _liststr = ["siteid=%s" % self.station, "siteAuthenticationKey=%s" % self.password ]
       
        # Go through each of the supported types, formatting it, then adding to _liststr:
        for _key in WOW._formats.keys() :
            v = record[_key]
            # Check to make sure the type is not null
            if v is not None :
                if _key == 'dateTime':
                    # For dates, convert from time stamp to a string, using
                    # the following format: YYYY-mm-DD HH:mm:ss, where ':'
                    # is encoded as %3A, and the space is encoded as either '+' or %20
                    v = urllib.quote(datetime.datetime.utcfromtimestamp(v).isoformat('+'), '-+')
                # Format the value, and accumulate in _liststr:
                _liststr.append(WOW._formats[_key] % v)
        # Add the software type and version:
        _liststr.append("softwaretype=weewx-%s" % weewx.__version__)
        # Now stick all the little pieces together with an ampersand between them:
        _urlquery='&'.join(_liststr)
        # This will be the complete URL for the HTTP GET:
        _url=self.http_prefix + _urlquery
        return _url

Alan Major
Birdham, UK

[Thomas Keffer]

Well done! An A+ for the student!

-tk

--
 
 

[Jason Pearce]

Hi,

Thanks for your efforts.

I have added class WOW to restul.py, but when it runs, I see the following error:

Jan 16 11:40:40 raspberrypi weewx[3770]: restful: Unrecoverable error when posting record 2013-01-16 11:40:00 GMT (1358336400) to WOW station 40024885

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****  'windGustDir'

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****  Traceback (most recent call last):

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****    File "/home/weewx/bin/weewx/restful.py", line 512, in run

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****      protocol.postData(self.archive, time_ts)

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****    File "/home/weewx/bin/weewx/restful.py", line 598, in postData

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****      _url = self.getURL(archive, time_ts)

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****    File "/home/weewx/bin/weewx/restful.py", line 644, in getURL

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****      v = record[_key]

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****  KeyError: 'windGustDir'

Jan 16 11:40:40 raspberrypi weewx[3770]:    ****  Thread terminating.

Jan 16 11:40:41 raspberrypi weewx[3770]: reportengine: Found configuration file /home/weewx/skins/Standard/skin.conf for report StandardReport

I assume this is because there is currently no wind ?

Thanks

--
 
 

[Jason Pearce]

Hi,

Fixed it :

I added windGustDir to the line archive_types

archive_types = ['dateTime', 'usUnits', 'barometer', 'outTemp', 'outHumidity',

                     'windSpeed', 'windDir', 'windGust', 'windGustDir', 'dewpoint', 'radiation', 'UV']

Regards,

Jason

[Thomas Keffer]

No, if there was no wind, the value of windGustDir would be 'None', rather than missing completely.

I suspect the problem is that while Alan provided a new interface to publish to WOW, he didn't notice that it takes one more data type than the standard 'Ambient' protocol on which it is based: 'windGustDir'. Ambient does not use the direction of wind gusts, so the weewx code doesn't bother retrieving it from the database. Hence, it's missing.

I can think of three solutions:

1. In addition to the methods Alan overrode, also provide a version of extractRecordFrom() which gets windGustDir as well.
2. Simply add windGustDir to list archive_types, which is the list of types retrieved by the default version of extractRecordFrom(). This means modifying the source code, but it seems like a reasonable thing to do, so I can add this to the source code base.
3. Watch for the error, ignore it, and keep going.

Of these, I like #2. I'll add windGustDir to the code base. It won't appear in a version for a while, so in the meantime you can simply patch your source code (below).

Index: bin/weewx/restful.py

===================================================================

--- bin/weewx/restful.py (revision 816)

+++ bin/weewx/restful.py (working copy)

@@ -40,7 +40,7 @@

     

     # The types to be retrieved from the arhive database:

     archive_types = ['dateTime', 'usUnits', 'barometer', 'outTemp', 'outHumidity', 

-                     'windSpeed', 'windDir', 'windGust', 'dewpoint', 'radiation', 'UV']

+                     'windSpeed', 'windDir', 'windGust', 'windGustDir', 'dewpoint', 'radiation', 'UV']

     # A SQL statement to do the retrieval:

     sql_select = "SELECT " + ", ".join(archive_types) + " FROM archive WHERE dateTime=?"  

 

-tk

[Alan Major]

Apologies, I should have mentioned that I had changed this part of the code too ... Sorry Jason!

Alan Major
Birdham, UK

[Jamie Dunbar]

Hello everyone.

So, to get this working I just need to insert the above code as an addition to restful.py?

Regards,

Jamie

[Alan Major]

Jamie

Yes, add the code in my earlier post plus the patch posted by Tom

Alan Major
Birdham, UK

[david reid]

Any chance of making the modified file available directly? Avoids issues of cut/paste. Will this be in future releases?

--
You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Thomas Keffer]

OK, how about a compromise. I've added the WOW protocol to restful.py, but I'm not going to put it in the configuration file. It's a little used protocol, so I don't want to clutter weewx.conf unnecessarily. 

I've implemented it slightly differently than Jason --- WOW is essentially the Ambient protocol with a few minor tweaks, so my implementation inherits directly from Ambient, thus saving a few lines of code.

I've enclosed a copy. To use it, replace your existing version of restful.py with it. Then add a section to your configuration file in the [StdRESTful] section that looks like:

[StdRESTful]

    .

    .

    .

    [[WOW]]

        

        #

        # This section is for configuring WOW posts

        #

        

        driver = weewx.restful.WOW

        station = 12345678

        password = foobar

I don't have a WOW station, so could you enthusiasts please test it?

-tk

[david reid]

Thanks. Will try it when next at home.

[peter.ho...@gmail.com]

Hi Tom

I have been monitoring this thread as I am setting up for a friend a WH1080 using weewx. It all seems to be working ok, with the odd lock up. But, others have reported that issue with this weather station.

I have now implemented your WOW changes, and they seem to be working ok. I will post back if we find any issues.

Thanks everybody for all the updates and you for the weewx system.

Peter Hothersall

[peter.ho...@gmail.com]

Forgot to mention that it running on a Raspberry Pi.

Peter Hothersall

[mwall]

On Friday, February 15, 2013 5:47:10 AM UTC-5, peter.ho...@gmail.com wrote:

I have been monitoring this thread as I am setting up for a friend a WH1080 using weewx. It all seems to be working ok, with the odd lock up. But, others have reported that issue with this weather station.

peter,

if you regularly experience USB lockups with a fine offset station, please try the ADAPTIVE polling mode to see if that helps. in weewx.conf, do this:

[FineOffsetUSB]
    model = 'WH1080'
    polling_mode = ADAPTIVE
    driver = weewx.fousb

there are now two polling modes for the fine offset stations: ADAPTIVE and PERIODIC.  periodic is the old behavior - weewx will poll the station every 30 seconds for data.  adaptive was added for 2.2.0 - weewx will attempt to poll the station when it is not busy reading the sensors or writing to memory.  note that periodic will take a bit longer to get started as it attempts to figure out console settings.

another thing you can do is to test the quality of the USB connection.  with weewx *not* running, run wee_config_fousb like this:

wee_config_fousb weewx.conf --check-usb

[Thomas Keffer]

These are features in the new V2.2 release. You'll have to install it to take advantage of them.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx Weather Station Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jim Woodruff]

I've added these steps but receive the following errors when trying to upload to WOW. According to their website I need to identify my pin #. I added it to the StdRESTfull section and still receive these errors.

Nov  7 09:45:43 freyr weewx[3939]: restful: Failed attempt #1 to upload to WOW
Nov  7 09:45:43 freyr weewx[3939]:    ****  Reason: HTTP Error 403: Forbidden
Nov  7 09:45:43 freyr weewx[3939]: genimages: Generated 16 images for StandardReport in 0.37 seconds
Nov  7 09:45:44 freyr weewx[3939]: restful: Failed attempt #2 to upload to WOW
Nov  7 09:45:44 freyr weewx[3939]:    ****  Reason: HTTP Error 403: Forbidden
Nov  7 09:45:44 freyr weewx[3939]: restful: Failed attempt #3 to upload to WOW
Nov  7 09:45:44 freyr weewx[3939]:    ****  Reason: HTTP Error 403: Forbidden
Nov  7 09:45:44 freyr weewx[3939]: restful: Failed to upload to WOW
Nov  7 09:45:44 freyr weewx[3939]: restful: Unable to publish record 2013-11-07 09:45:00 PST (1383846300) to WOW station XXXXXXXXXX
Nov  7 09:45:44 freyr weewx[3939]:    ****  Failed upload to site WOW after 3 tries

How do I accomplish this?

Jim

[Thomas Keffer]

No idea. I'm assuming you have the right station ID and password?

-tk

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.

[Frank Johnson]

any update on posting to WOW i have a pin it requires but dont know where to put it...

all i get is bad login...

   Frank

KF4CQ

[Thomas Keffer]

The directions should be in your configuration file weewx.conf. It's pretty simple:

[StdRESTful]

 ...

  [[WOW]]

    station = your WOW station ID

    password = your WOW password

-tk

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Frank Johnson]

Guess im getting a little slow to post to WOW you must create a user and password then a site 

that gives you a site id then you crate a 6 dig pin that becomes your password for posting wew 

it only took 4 hours to figure that out...

 thanks for help 

hope this helps others

 Frank

KF4CQ

wow site 899366001

[Ξ]

Earlier today I signed up for WOW and I entered my site ID and the 6 digit pin in:

[StdRESTful]

 ...

  [[WOW]]

    station = site ID

    password = PIN

but I get "restx: WOW: bad login; waiting 60 minutes then retrying", I also tried with my password for my WOW account, but that doesn't work either. Any clues?!
I'm using weewx 3.0.1

[Messy Potamia]

My WOW station id is a 9-digit account they emailed me, and the pin a 6-digit numeric pin. I seem to recall it didn't take right away, but then I restarted weewx a while later and it began publishing data. /hth/
Phil

[Ξ]

I've tried restarting weewx/R-Pi, but it still doesn't work. The 9-digit station ID is assigned when you create a new site. Not sure if relevant but what I also saw in the log each time when shutting down weewx is:


systemd[1]: Stopping weewx...
weewx[152]: engine: Received signal TERM.
weewx[152]: engine: Shut down StdReport thread.
weewx[152]: ws28xx: RFComm: handleHistoryData: next=None
weewx[152]: restx: Unable to shut down WOW thread
weewx[152]: restx: Shut down PWSWeather thread.
weewx[152]: restx: Shut down Wunderground-PWS thread.
weewx[152]: restx: Shut down StationRegistry thread.
weewx[152]: ws28xx: MainThread: stopRFThread: waiting for RF thread to terminate
weewx[152]: ws28xx: RFComm: stopping rf communication
weewx[152]: ws28xx: MainThread: releasing USB interface

[Petchara Inthanon]

I can use it 

Thank you So much

เมื่อ วันอังคารที่ 10 กุมภาพันธ์ ค.ศ. 2015 23 นาฬิกา 07 นาที 12 วินาที UTC+7, Ξ เขียนว่า:

[John Devine]

I just setup an account at WOW and installed weewx tonight and enabled the relevant section in weewx.conf and it just worked with sample data

JD

[jon.bar...@gmail.com]

Guys - I know a pretty old thread...but I recently had my MET account upgraded to the new MET office WOW.

Although I migrated my account, I had not connected Weewx to it. so now Im doing so.

the account migrated fine, and I had to 'create a site' - which I did.  I see there is a long unique ID for the site itself ) like (c58da0c0-0729-f021-9d47-6027bddefb8a) -  I just created it

but where do I find the 'password' - or as descirbed in the Weewx docs the 'six digit numerical pin' - as I dont see this in the site...the only other thing I see is a 'user id' (similar format to the long unique site Id) - but using that, weewv complains...

Any advice ?

[Jimi Lawson]

The password is called an Authentication Key in WOW. You create it yourself in WOW by logging into your site then select edit site,  you add a key in there.
"WOW provides a simple API for you to automatically upload data into WOW. The easiest solution for most users is to use a piece of software that supports this API, and some of these are listed below. If you would prefer not to use these or are writing your own, please see the details further down the page about the API.

Before you attempt to upload you will need 2 numbers. The first is your Site ID. This is an arbitrary number that is used to distinguish your site from another. This number appears (in brackets) next to or underneath the name of your site on the site information page, for example (956916003) for pre-June 2016 sites or (6a571450-df53-e611-9401-0003ff5987fd)

The second item you require is the Authentication Key. This is a 6 digit number that is used to ensure data is coming from you and not another user. If you do not know (or have not created) the Authentication Key, go to the Edit pages of any site you own and the value is an option under the Site Details section. Create and enter a 6-digit numeric PIN of your choice (it is not supplied by the kit manufacturer). The Authentication Key may also be referred to as the 'AWS Pin'.

You must take note of both these numbers before continuing, otherwise you will not be able to upload weather data to WOW. "
Regards
Jimi

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/97fb053b-ba2f-468b-9ac8-d6e986465c66n%40googlegroups.com.

[jon.bar...@gmail.com]

Thanks Jimi - I had read the Weewx materials....

but yes, it was an oversight on my part...when creating ...or editting a Met Office WOW site, it now asks for 'authentication key' as one of the many fields to setup the site itself.

I had obviously had a brain freeze during once setup - and couldnt see that field unless going into the 'edit' for the site.

One change to the Weewx docs though: the key is no longer a six digit pin (which probably contributed to my brain freeze on the topic) and you can enter any text you want as a key.

My site is now up and puslishing to MET via the migrated WOW account I have !

Jon.

PS Weewx is still the greatest software ! it just keeps giving and giving !

[Jimi Lawson]

No problem Jon, glad I was able to help. Your correct about the six digit pin, it hasn't been that for a long time, mine has been 8 characters since at least 2016.
Regards
Jimi

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/55e74958-7336-43d0-9fb1-c64fb341bd69n%40googlegroups.com.