Failure of wunderfixer

161 views
Skip to first unread message

Gareth Hunt

unread,
Jun 7, 2019, 8:23:29 PM6/7/19
to weewx-user
Greetings,

[Still a newbie user.]

I live in rural USA where a landline DSL will probably be available some time after Hell freezes over.  So I have to rely on a satellite link.  Not surprisingly, this fails when there are adverse atmospheric conditions.  There were thunderstorms for several hours on June 7.  So wunderfixer to the rescue!  However, it failed.  Here is the code snippet with my added print statement in class Wunderstation.

        _url = "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=%s&"\
        "month=%d&day=%d&year=%d&format=1" % (self.station, dayRequested_tt[1], dayRequested_tt[2], dayRequested_tt[0])
        print _url   # My addition
       
        try :
            # Hit the weather underground site:
            _wudata = urllib2.urlopen(_url)
        except urllib2.URLError as e :
            print >>sys.stderr, "Unable to open Weather Underground station " + self.station, " or ", e

This gave:

bin$ ./my.wunderfixer
Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KVASCOTT14&month=6&day=7&year=2019&format=1
Unable to open Weather Underground station KVASCOTT14  or  HTTP Error 403: Forbidden
Could not get Weather Underground data. Exiting.

However, when I copied and pasted the printed http address into my browser, everything was fine.  I could see all of the reported records with the expected data gaps.  I did not get HTTP Error 403 (i.e., e).

So I have some (hopefully trivial) permission error with my access to Wunderground.  Can somebody suggest what it is and how I get it fixed?

Thanks,
Gareth.

Sean Kirkpatrick

unread,
Jun 7, 2019, 8:54:50 PM6/7/19
to weewx-user
Hi Gareth.

I'm seeing the same behavior and have been for over a week now. The problem seems to be with WeatherUnderground itself (not surprising!)

   Sean

Sean Kirkpatrick

unread,
Jun 11, 2019, 7:43:30 PM6/11/19
to weewx-user
Hi Gareth.

I've been puzzled by this and have been doing some debugging. I started with reproducing your code above in my own copy of wunderfixer. I began by trying to see what the actual contents of the error is, and that led me to realize that the exception handler in that section of code isn't where the exception is being thrown. If you look in the vicinity of line 207, you should see something like the following

   try:
        # Get all the time stamps on the Weather Underground for the given day:
        wunder_results = wunder.getDayTimeStamps(date_date)
    #except Exception as e: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I commented out
    except  urllib2.HTTPError, error: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I added
# here
        contents = error.read() <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I added
        print contents               <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I added
        wlog.slog(syslog.LOG_ERR, "Could not get Weather Underground data. Exiting.")
        exit("Could not get Weather Underground data. Exiting.")

This causes the actual web page (the HTML content that describes the actual error) to be printed on the console. in my case, the error looks like this

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;www&#46;wunderground&#46;com&#47;weatherstation&#47;WXDailyHistory&#46;asp&#63;" on this server.<P>
Reference&#32;&#35;18&#46;3f75d917&#46;1560296045&#46;2172682b
</BODY>
</HTML>

Could not get Weather Underground data. Exiting.

What this tells me is that something is wrong with my station configuration. And yet, my periodic posts are sent correctly to WU. Hmmmm. 

I know that WU was rebuilt not long ago. I wonder if that might be part of the problem. Unfortunately, I'm not much of  python guy so I'm rather on the slow side in debugging.

I'll post more as I find it.

 
 

On Friday, June 7, 2019 at 5:23:29 PM UTC-7, Gareth Hunt wrote:

Sean Kirkpatrick

unread,
Jun 11, 2019, 8:00:38 PM6/11/19
to weewx-user
Continuing saga.

the actual body of the error (would be good for weewx to log this, i think) says

Access Denied

You don't have permission to access "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?" on this server.

Reference #18.3f75d917.1560296691.218110b4


looking at that URL, it doesn't look right. the ? at the end is looking for a set of parameters to be passed to the web server hosting WU. google turned up the following example
 

this gave me the format for modifying the URL for my own station

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCASONOM43&format=XML

voila! I now get that data in my browser. so it seems as if wunderfixer isn't properly building the URL string for some reason.

my station is being powered by a Raspberry Pi. they are notorious for chewing up SD cards which have a limited number of writes before they start to go bad. i've had problems before where the station behavior was very strange, and I strongly suspect that this might be the case now. i think i'll rebuild the OS on a new SD card and see what the behavior is.

in the meantime, you can use the --verbose option on the command line to see some additional details as wunderfixer runs.

     Sean
 

Sean Kirkpatrick

unread,
Jun 11, 2019, 8:37:11 PM6/11/19
to weewx-user
Argh! I've pretty much confirmed that my SD card is toast. Getting lots and lots of errors trying to run yum update as well as wunderfixer - 

seank@weather:~ $ wunderfixer --verbose
Traceback (most recent call last):
  File "/usr/bin/wunderfixer", line 64, in <module>
    import weecfg
  File "/usr/share/weewx/weecfg/__init__.py", line 28, in <module>
    import weewx.defaults
  File "/usr/share/weewx/weewx/defaults.py", line 227, in <module>
    defaults = configobj.ConfigObj(StringIO(default_str))
  File "/usr/lib/python2.7/dist-packages/configobj.py", line 1229, in __init__
    self._load(infile, configspec)
  File "/usr/lib/python2.7/dist-packages/configobj.py", line 1318, in _load
    raise error
configobj.ConfigObjError: Parsing failed with several errors.
First error at line 112.

Rebuilding my Pi...again...sigh!

   Sean

On Friday, June 7, 2019 at 5:23:29 PM UTC-7, Gareth Hunt wrote:

gjr80

unread,
Jun 12, 2019, 3:09:47 AM6/12/19
to weewx-user
SD card issues aside the current wunderfixer will almost certainly not work due to WU changes. As you have discovered wunderfixer currently uses a call to WXDailyHistory.asp to obtain the days records held by WU and wunderfixer uses this to determine which records to re-upload, WU seems to be decommissioning WXDailyHistory.asp and consequently a call to WXDailyHistory.asp will almost certainly always fail causing wunderfixer to fail. There is some work going on to try to update wunderfixer to use the new WU API; however, the new API has some issues that either need to be fixed by WU or some substantial changes made to wunderfixer.

So for the time being it is a case of standby.

Gary

Sean Kirkpatrick

unread,
Jun 12, 2019, 11:44:55 AM6/12/19
to weewx-user
Standing by...

I guess that I didn't search hard enough to find out about the new work being done. Thanks Gary and Leon for correcting my course!

    S


On Friday, June 7, 2019 at 5:23:29 PM UTC-7, Gareth Hunt wrote:

Gareth Hunt

unread,
Jun 12, 2019, 1:43:22 PM6/12/19
to weewx-user
Hi Sean,

Thanks for looking into this.  Like you, I did not know of the work in progress when I posted.  Leon pointed me to a (temporary) fix of the old wunderfixer code, while he struggles with the new WU API.  This fix also has a new timeout option.  Even setting --timeout=30 did not succeed initially, but I have used it successfully since.  I also put this version in the recommended nightly cron job at 23:58.  It has worked most there most of the time, but I'm still checking it each morning.

I'm also Standing by ...

Cheers,
Gareth.

Sean Kirkpatrick

unread,
Jun 14, 2019, 9:35:58 PM6/14/19
to weewx-user
Hi Gareth.

I finally had a chance to d/l the fixed version of Wunderfixer from github. Seems to work just fine for me. More testing to be done...

    Sean

PS: Thanks Leon!


On Friday, June 7, 2019 at 5:23:29 PM UTC-7, Gareth Hunt wrote:
Reply all
Reply to author
Forward
0 new messages