We have finally put the last nail on the coffin for Wunderfixer?

1,003 views
Skip to first unread message

tds

unread,
May 21, 2019, 2:09:14 PM5/21/19
to weewx-user
For the last three days, my daily cron job of wunderfixer have returned "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" (May 20).

Jarom Hatch

unread,
May 21, 2019, 3:46:31 PM5/21/19
to weewx-user
I'm getting 403.  I entered an issue but upon looking further it appears to me that wunderground has either broken that url or intentionally deactivated it.  Hoping it isn't dead forever.

Thomas Keffer

unread,
May 21, 2019, 4:33:46 PM5/21/19
to weewx-user
It looks like the WU completely changed their API without warning. For example, to access my station, it used to be
https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KORHOODR3

but now it's

https://www.wunderground.com/dashboard/pws/KORHOODR3

You used to be able to download a day's worth of data, but now I'm not sure how to do that without screen scraping. Wunderfixer may no longer be possible to do, at least not accurately


-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/54718d99-b68d-4f34-8213-4786a2578265%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jarom Hatch

unread,
May 21, 2019, 5:59:35 PM5/21/19
to weewx-user
Just tried posting a question to WU using their email form.  Got this error:  

Validation failed: weather_underground is invalid.

Looks like they're a mess right now.

Leon Shaner

unread,
May 21, 2019, 7:52:37 PM5/21/19
to weewx...@googlegroups.com
FWIW, I had the odd 404 the other day, but then hours later it worked.
They may be just moving infrastructure around and not getting all the load balancer rules correct on the first go. Hope that's all it is...

Could it be that we need to request an API key and use that as a password, instead?

Here is what I see:

/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Thu 16 May 13:17:01 EDT 2019 Reason: HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Fri 17 May 13:17:01 EDT 2019 Reason: HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log.1:Sat 18 May 03:02:03 EDT 2019 Reason: HTTP Error 404: Not Found
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Mon 20 May 13:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 01:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Unable to open Weather Underground station KMIDEARB5  or  HTTP Error 403: Forbidden
/usr/share/weewx/log/weewx.log:Tue 21 May 13:17:01 EDT 2019 Reason: HTTP Error 403: Forbidden


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

gjr80

unread,
May 21, 2019, 11:02:24 PM5/21/19
to weewx-user
Wunderfixer (as it stands now) relies on WUs WXDailyHistory.asp page. If WXDailyHistory doesn't work then wunderfixer will not work. Period.

I guess depending on where you sit on the conspiracy theory scale will determine whether you believe the current issues with WXDailyHistory are short term glitches or a silent closure of the service. There are a few threads on wxforum.net that touch on this issue, and of course many others that touch on the WU issues over the last not so long period. There seems to have been a few posts from WU staff indicating that 'all the old stuff is very fragile', so that could mean short term glitch, but on the other hand WU has been pushing folks towards their 'new' API which could mean the end for WXDailyHistory.

The 'new' WU API that is available to PWS owners/uploaders does have a '1 day rapid history request' that provides what seems like the same sort of info as WXDailyHistory, albeit in JSON format and with a whole pile more cruft, well at least as far as wunderfixer is concerned. So in theory there is no reason wunderfixer could not be reworked to utilise the '1 day rapid history request' data. Of note to use the new API you need to have a (new) API key, whereas WXDailyHistory/wunderfixer does not require an API key. One shortcoming I did notice with the '1 day rapid history request' is there appears to be no way to obtain a given days data; you get the current day only, whereas with WXDailyHistory you can choose the date required, so that would mean no more historical wunderfixing.

In any case, those who may consider WU as a de facto backup may want to start brushing up on their bash/MySQL skills and implement a robust local backup arrangement (though on WUs form you should have done that ages ago).

Gary

gjr80

unread,
May 21, 2019, 11:53:21 PM5/21/19
to weewx-user
Hmm, I notice that WXDailyHistory is working as usual now (well for me anyway) but wunderfixer still gives a 403 error.

Gary

Leon Shaner

unread,
May 22, 2019, 9:35:18 AM5/22/19
to weewx...@googlegroups.com
For one thing, the URL of this form:


Is now redirecting to one using HTTPS:


Also, the redirect itself takes an excruciatingly long time.
So I just changed the URL to https directly...

The first time I tried any of the above using CURL this morning it worked, but then after that I started getting:

An error occurred while processing your request.

Reference #30.6f451160.1558531514.16ced4f6

It looks as if they've put some kind of Akamai proxy in the middle, which is fine for static content, but not so fine for a query of this nature.  Strange that it worked for me the very first time.  It's almost as if the Akamai "farm" has lost some "state" information and not all nodes have the same content, so if you get stuck going through a bad node you get a bogus response.

Attached is a transcript of a failed attempt.  I put SOMESTATION there only after the fact.  The actual query was for my actual station, which used to work.

curl.txt

Jarom Hatch

unread,
May 22, 2019, 1:48:08 PM5/22/19
to weewx-user
I was able to get it to work twice in my web browser, but as you said, it is sporadic.  I don't ever recall them using Akamai before so that may very well be a contributing factor.

I wonder if we can find out the origin address and see what happens if we can bypass Akamai...

Jarom Hatch

unread,
May 22, 2019, 2:04:22 PM5/22/19
to weewx-user
Interesting, using curl sometimes I can it fine, but wunderfixer is always getting a 403 Forbidden, as if it is actively being blocked...  When it doesn't work in curl I get `HTTP/1.1 404 Not Found` and when it does work I get `HTTP/1.1 200 OK`.  Curl never gets a 403 error.

Leon Shaner

unread,
May 22, 2019, 2:42:53 PM5/22/19
to weewx...@googlegroups.com
Jarom,

CURL is pretty sophisticated in its ability to emulate browser state in pretty much any way but JavaScript.  When it worked this morning, I saw some cookies were involved.
It may well be that the python way isn't handling that part.
I don't know enough about how python fetches pages to work that out, but I am very familiar with CURL, so if I can find a path that works consistently, then I'll go back to the python to see about how to implement same.

I was getting 404's in the browser even, when I looked at it earlier.

I'll keep working on it, but not too hard, so as to not get on their radar in any unwanted sort of way.  ;-)

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Leon Shaner

unread,
May 22, 2019, 4:20:37 PM5/22/19
to weewx...@googlegroups.com
I'm still working on this.
CURL is telling me they are not only using https, but also TLSv1.2.
Here is a transcript, in case one of y'all beats me to the fix.  =D

wu.txt

Doug Bo

unread,
May 22, 2019, 4:33:20 PM5/22/19
to weewx-user
Sort of a noob question or maybe a feature request: how about adding a web interface to search the database for historic weather data?  I grab the wunderground data that I have uploaded to add to various farm records a few months or even a year after the fact.  If WU is getting worse perhaps I need another solution?

Thanks,
Doug B.

Leon Shaner

unread,
May 22, 2019, 4:41:08 PM5/22/19
to weewx...@googlegroups.com
Hey, Doug,
Creating a front end to search your own local data would certainly be doable.
For sure, directing end-users to our own sites and letting them search our historical data would be useful.

But that wouldn't fix the problem at hand.  :-/
The wunderfixer utility is there to query what data WU has on record from your station and to re-upload any missing data.
Basically if we can't fix this then it means WU will often be missing data from time to time for various reasons.
Really only WU should care and end-users who go to their site to look at historical data.   So yeah, they should be invented to help, since they're the ones missing the data, and the less reliable their data, the more likely even end-users will begin to abandon using them.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Doug Bo

unread,
May 22, 2019, 5:06:56 PM5/22/19
to weewx-user
Hi Leon,

Agreed, I realize my post is a bit off topic but if WU continues to be less than reliable then perhaps other solutions might be viable?
Thanks for the reply, perhaps a new thread needs to be started by someone smarter than me about historical data search.

Good luck with your research & fix.  I'm a big WU fan and am sad they are slipping.  Thank you for trying to resolve the issue.  :) 

Thanks again,
Doug B.

On Wednesday, May 22, 2019 at 1:41:08 PM UTC-7, Leon Shaner wrote:
Hey, Doug,
Creating a front end to search your own local data would certainly be doable.
For sure, directing end-users to our own sites and letting them search our historical data would be useful.

But that wouldn't fix the problem at hand.  :-/
The wunderfixer utility is there to query what data WU has on record from your station and to re-upload any missing data.
Basically if we can't fix this then it means WU will often be missing data from time to time for various reasons.
Really only WU should care and end-users who go to their site to look at historical data.   So yeah, they should be invented to help, since they're the ones missing the data, and the less reliable their data, the more likely even end-users will begin to abandon using them.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On May 22, 2019, at 4:33 PM, Doug Bo <dou...@gmail.com> wrote:

Sort of a noob question or maybe a feature request: how about adding a web interface to search the database for historic weather data?  I grab the wunderground data that I have uploaded to add to various farm records a few months or even a year after the fact.  If WU is getting worse perhaps I need another solution?

Thanks,
Doug B.

On Tuesday, May 21, 2019 at 11:09:14 AM UTC-7, tds wrote:
For the last three days, my daily cron job of wunderfixer have returned "503 Unavaialbe" (May 18), "404 Not Found" (May 19) and "403 Forbidden" (May 20).

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

Leon Shaner

unread,
May 22, 2019, 7:25:03 PM5/22/19
to weewx...@googlegroups.com
Hey WeeWX'ers!!!  =D

I have a fix in the hopper.

There's nothing that can be done for the occasional HTTP 404, or even 503's I am now seeing, but the HTTP 403 was due to a change on WU's part where they are rejecting certain HTTP User-Agent strings.  The fact that they are putting Akamai in the middle is almost certainly a great thing re: their scalability issues; however, they probably inherited some default settings that filter "bots" and malware and such, which is likely why the HTTP User-Agent now matters.

I have set the User-Agent to "CURL" and it works.
I have set it to "Mozilla" and it works.  I'm going with that one, since it means Mosaic Killer, both of which were among the the very first User-Agents I ever worked with, circa 1993 back before there was such as thing as Netscape.  =D

/ye-olde-farte mode off  ;-)

My testing has so far been under Python3, but coincidentally (and not a causation), WU started throwing HTTP 503's around the time that I tried validating my code also under Python2.

Everything is working against today's date.
It's when I go after yesterday's date that I get the HTTP server error 503.

I expect the 404's and 503's to go away eventually, but at least for now I have a fix for the 403 (forbidden)'s, just based on the User-Agent string.

I'll submit a change for wunderfixer both to the 3.9.x "master" and 4.0.x "development" branches in a moment and reply back with direct links for anyone who wants a fix sooner.  =D

Isn't this fun?  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
<wu.txt>



Working from here:
https://docs.python.org/2/library/ssl.html

So far I have tried this, to no avail.
Really just doing the "import ssl" and using https in the URL, and adding context=ssl_context to the urllib.request.

A snippet of that looks as follows, but still getting 403 forbidden.  :-(

# For new WU interface which uses SSL and TLSv1.2
import ssl

...

               "&month=%d&day=%d&year=%d&format=1" % (self.station, dayRequested_tt[1],
                                                      dayRequested_tt[2], dayRequested_tt[0])

        # specify TLSv1.2 and SSLv2, but not SSLv3
        ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
        ssl_context.options |= ssl.PROTOCOL_SSLv23
        ssl_context.options |= ssl.OP_NO_SSLv3

        try :
            # Hit the weather underground site:
            _wudata = urllib.request.urlopen(_url, context=ssl_context)



Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
May 22, 2019, 9:03:49 PM5/22/19
to weewx...@googlegroups.com
Say, we need a tester who is still on 3.9.1 or there abouts to try this out:

https://raw.githubusercontent.com/UberEclectic/weewx/master/bin/wunderfixer

Can't do anything to workaround WU's sporadic 404 and 503 errors, but at least the 403 error should be gone.

I was able to test the 4.0 / development version myself on both Python2 and 3, so hopefully Tom will merge that soon.  It's over here if you are impatient.  =D



Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Jarom Hatch

unread,
May 23, 2019, 1:57:47 AM5/23/19
to weewx-user
I tried the 3.9.1 version and I get Could not get Weather Underground data. Exiting.

Curl still works, even for yesterday's data.  Tracing the script it doesn't appear to be actually attempting the download.  

Hey WeeWX'ers!!!  =D
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.
<wu.txt>
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

Leon Shaner

unread,
May 23, 2019, 10:27:37 AM5/23/19
to weewx...@googlegroups.com
Jarom,

Thanks so much!  I see what I did wrong and I was able to make a stubbed down version for basic testing to prove it's at least trying to connect.

Same location (for WeeWX 3.9.1):


The thing is, I'm still constantly getting 404 (Not Found) even with CURL, and just a bit ago the site started throwing 503 (Service Not Available).
So...  It's kinda hard to test under these conditions.   But as long as you don't get a 403, then at least my User-Agent "hack" will be "proven."   :-/


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/49f5ecce-f082-4d64-848c-98e07e3d6349%40googlegroups.com.

Jarom Hatch

unread,
May 23, 2019, 10:59:45 AM5/23/19
to weewx-user
Sad, now I'm getting the 503.  I'll keep trying.

Jarom Hatch

unread,
May 23, 2019, 11:06:55 AM5/23/19
to weewx-user
Those 503's are coming from Akamai.  When I have used Akamai in the past those errors are because the origin is not responding correctly.

Related sorta, when I took out the code to pull the timestamps and just force-ran all my records as a big backfill I got success messages back however none of the records actually made their way in.

Leon Shaner

unread,
May 23, 2019, 12:10:28 PM5/23/19
to weewx...@googlegroups.com
Jarom,

So you got 503's with my updated code, but not 403's?

Yeah, I have found that it is normal for WU to lazily process re-uploaded records, and it seems as if they drop them...as if some backend process is dying.

I found through trial and error that after a weewx "outage" I need to re-upload exactly 10 times, but not back to back.  I do one upload every 20 minutes, which means spreading the 10 re-uploads out over a 200 minute period.
After that wunderfixer reports no missing records.
Well, at least it used to work before all these 404's and 503's.  :-(


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/db878ac7-8255-4704-9e69-051647ae8f98%40googlegroups.com.
Message has been deleted

Jarom Hatch

unread,
May 23, 2019, 2:35:00 PM5/23/19
to weewx-user
I haven't seen a 403 since you changed the user agent.


On Thursday, May 23, 2019 at 10:10:28 AM UTC-6, Leon Shaner wrote:
Jarom,

Jarom Hatch

unread,
May 23, 2019, 2:48:08 PM5/23/19
to weewx-user
Yesterday when I was able to download the csv I saved a copy of it.  I put it up on a site hosted by me and changed the URL in wunderfixer to point there instead.  Now it parses it and starts the backfill.  Still isn't showing up on their side though.  I'm going to run it about 10 times over the next few hours and see what happens.  Could be that Akamai is blocking the backfill.  Or they have totally jacked up their backend.

Jarom Hatch

unread,
May 23, 2019, 9:22:18 PM5/23/19
to weewx-user
Even after that none of the backfilled data made it in. If they are no longer accepting backfilled data then fixing wunderfixer might not be possible.

Leon Shaner

unread,
May 23, 2019, 9:33:30 PM5/23/19
to weewx...@googlegroups.com
I am in contact with IBM. This whole intersection is entirely for THEIR benefit.
I am persistent SOB. It'll be fixed. LOL

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 23, 2019, at 9:22 PM, Jarom Hatch <jsh...@gmail.com> wrote:
>
> Even after that none of the backfilled data made it in. If they are no longer accepting backfilled data then fixing wunderfixer might not be possible.
>
> --
> You received this message because you are subscribed to the Google Groups "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/76f9aa83-37ac-4014-aa04-7f675d63a6e1%40googlegroups.com.

Leon Shaner

unread,
May 24, 2019, 1:35:10 PM5/24/19
to weewx...@googlegroups.com
FYI, all, I am currently hot on the trail to converting the wunderfixer query to use the new API instead.  The new API is going to be much easier than the old query, because the output is neatly formatted as JSON, and crucially for each record they even include the "epoch" i.e. Unix timestamp, which is all we care about in that query anyway.
The use of "epoch" means I won't even have to convert the date strings to do the matching.  =D
Even the input date parameter is cleaner a la 20190524.  =D

What I am slightly undecided about is how/where to store the API key.
I think for compatibility reasons I'll add a new field in the weewx.conf for the API key.
That way existing code based on a password can still use that method, and over time newer code can use the API key explicitly instead of a password.

Once I have the new wunderfixer working, for weewx 4.0, I'll work to backport the changes to weewx 3.9.1.

Hopefully I don't forget also to update wee_debug to obfuscate the API key, before I'm done with all of this.  LOL

Docs on the new API are over here...  This is for people like us who have a PWS, as opposed to other people who have to pay for API usage:

The API Key is generated in the "Member Settings" section of wunderground.com ("My Profile" > "Member Settings" > "API Keys").


With any luck I'll have this banged out in a few hours.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On May 23, 2019, at 9:33 PM, Leon Shaner <le...@isylum.org> wrote:

I am in contact with IBM.  This whole interface is entirely for THEIR benefit.

Leon Shaner

unread,
May 24, 2019, 5:01:19 PM5/24/19
to weewx...@googlegroups.com
Hi, WeeWX'ers!  =D

For those on WeeWX 4.0 / development, ONLY...
I am done with changing wunderfixer to use the new wunderground API KEY way of doing the WU query to see what records they have for comparison to those in the archive DB.   Yay!  =D

Please see the comments in the pull request over here:


And until that gets merged you can grab a copy from here -- again this is the 4.0 / development version, ONLY:

Fun fun!  =D


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
May 24, 2019, 6:30:29 PM5/24/19
to weewx...@googlegroups.com
Well shoot.

Found an issue that surfaced only upon weewx restart.
Don't add the apiKey to weewx.conf.
Found out engine.py is picky about properties it isn't expecting to find in the weewx.conf file.  Thought it would just ignore them.
So I guess I need to extend the dictionary of expected properties now, too.  :S

But you can still run this new wunderfixer on the side by passing the --apikey argument.

You could for example just call it wunderfixer_apikey and always pass your key on command-line:

$ wunderfixer_apikey --apikey=1234567890123456789012345678901
(With your real key there)...

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

gjr80

unread,
May 24, 2019, 7:29:01 PM5/24/19
to weewx-user
What does the error trace show? WeeWX shouldn't be concerned about extra config options. We don't need to re-invent wheels, forecast extension (among others) already uses a WU API key, not appropriate to use that config option here but using a consistent config option naming convention makes it easier for users.

Gary

Leon Shaner

unread,
May 24, 2019, 8:09:45 PM5/24/19
to weewx...@googlegroups.com
Hey, Gary,

Below is the error on startup, simply because I added the "apiKey = blah" right after the "password = oog" in the [[Wunderground]] section.

This led me to believe that I need to somehow "extend" the ambient_dict, but for the life of me I am not seeing how to do that. I looked every where for station and password in the restx.py and weecfg/__init__.py, and so far I am absolutely stumped.

Of course I appended to what seemed like an obvious place in restx.py, but nope. No good:

_ambient_dict = get_site_dict(
config_dict, 'Wunderground', 'station', 'password', 'apiKey')
if _ambient_dict is None:
return

Same exact error without or with that added. :-(

Any insights very much appreciated. =D

May 24 19:59:03 nixie weewx[17779]: restx: WU essentials: {}
May 24 19:59:03 nixie weewx[17779]: engine: Caught unrecoverable exception in engine:
May 24 19:59:03 nixie weewx[17779]: **** __init__() got an unexpected keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]: **** : Traceback (most recent call last):
May 24 19:59:03 nixie weewx[17779]: **** : File "/usr/share/weewx4/bin/weewx/engine.py", line 892, in main
May 24 19:59:03 nixie weewx[17779]: **** : engine = engine_class(config_dict)
May 24 19:59:03 nixie weewx[17779]: **** : File "/usr/share/weewx4/bin/weewx/engine.py", line 79, in __init__
May 24 19:59:03 nixie weewx[17779]: **** : self.loadServices(config_dict)
May 24 19:59:03 nixie weewx[17779]: **** : File "/usr/share/weewx4/bin/weewx/engine.py", line 143, in loadServices
May 24 19:59:03 nixie weewx[17779]: **** : self.service_obj.append(weeutil.weeutil._get_object(svc)(self, config_dict))
May 24 19:59:03 nixie weewx[17779]: **** : File "/usr/share/weewx4/bin/weewx/restx.py", line 607, in __init__
May 24 19:59:03 nixie weewx[17779]: **** : **_ambient_dict)
May 24 19:59:03 nixie weewx[17779]: **** : TypeError: __init__() got an unexpected keyword argument 'apiKey'
May 24 19:59:03 nixie weewx[17779]: **** Exiting.
May 24 19:59:03 nixie systemd[1]: weewx.service: Main process exited, code=exited, status=1/FAILURE
May 24 19:59:03 nixie systemd[1]: weewx.service: Unit entered failed state.
May 24 19:59:03 nixie systemd[1]: weewx.service: Failed with result 'exit-code'.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

> On May 24, 2019, at 7:29 PM, gjr80 <gjrod...@gmail.com> wrote:
>
> What does the error trace show? WeeWX shouldn't be concerned about extra config options. We don't need to re-invent wheels, forecast extension (among others) already uses a WU API key, not appropriate to use that config option here but using a consistent config option naming convention makes it easier for users.
>
> Gary
>
> --
> You received this message because you are subscribed to the Google Groups "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b069c856-8c59-4864-b724-e5f88ca6908e%40googlegroups.com.

gjr80

unread,
May 25, 2019, 1:24:42 AM5/25/19
to weewx-user
No, all get_site_dict does is check that the listed config options exist and are not the install defaults. The real issue is that _ambient_dict is used to pass keyword arguments to class AmbientThread init, apiKey is not a parameter in the init signature so hence the error. One approach will be to simply add an apiKey=None parameter to the AmbientClass init signature, it won't be used but who knows where WU may end up, maybe it could be thought of as a bit of future proofing/planning. Another approach would be to have a separate [Wunderfixer] stanza in weewx.conf, seems a waste to me and it makes sense to have the WU parameters together in the one place. Maybe leave it as a command line parameter only for wunderfixer, to me that seems almost the same as having a separate [Wunderfixer] stanza. Maybe introduce whatever approach in 4.0 and leave it as a wunderfixer command line parameter for the (expected) 3.9.2 release. Probably one for guidance from Tom.

Gary

Leon Shaner

unread,
May 25, 2019, 8:08:43 AM5/25/19
to weewx...@googlegroups.com
Hey, Gary,

I figured this out last night with a cue from Tom. But it requires modifying two different classes in restx.py, similar to the way you mentioned and I am a little uncomfortable with it, since I don't know all the different touch points / ramifications for other code.

Also, something strange started happening toward the end last night and the list of missing records grew to pretty much 100%.
I double checked and the new query is still pulling back the expected ~288 records (every 5 mins, is 12 an hour, so 288 in a day), but I now need to scrutinize the "epoch" values, because it's almost as if IBM changed them by some offset in the middle of the day, so now none of them "line up" with my local data.

I am heading out of town, so I will have to pick this up on Monday.

Thanks for the help! =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPhone)

> On May 25, 2019, at 1:24 AM, gjr80 <gjrod...@gmail.com> wrote:
>
> No, all get_site_dict does is check that the listed config options exist and are not the install defaults. The real issue is that _ambient_dict is used to pass keyword arguments to class AmbientThread init, apiKey is not a parameter in the init signature so hence the error. One approach will be to simply add an apiKey=None parameter to the AmbientClass init signature, it won't be used but who knows where WU may end up, maybe it could be thought of as a bit of future proofing/planning. Another approach would be to have a separate [Wunderfixer] stanza in weewx.conf, seems a waste to me and it makes sense to have the WU parameters together in the one place. Maybe leave it as a command line parameter only for wunderfixer, to me that seems almost the same as having a separate [Wunderfixer] stanza. Maybe introduce whatever approach in 4.0 and leave it as a wunderfixer command line parameter for the (expected) 3.9.2 release. Probably one for guidance from Tom.
>
> Gary
>
> --
> You received this message because you are subscribed to the Google Groups "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/68bfe21d-d0a2-4c55-9e71-2d65a69eea03%40googlegroups.com.

Leon Shaner

unread,
May 25, 2019, 8:54:25 AM5/25/19
to weewx...@googlegroups.com
So this is where I am...
These all seem reasonable (but then scroll for the problem which starts with yesterday's date):

pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-21 --test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-21
Number of archive records:     1440
Number of WU records:          288
Number of missing records:     4

Missing records:
2019-05-21 00:00:00 EDT (1558411200); 29.379";  49.8F;  78%; 0.0 mph; N/A deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
2019-05-21 00:01:00 EDT (1558411260); 29.379";  49.7F;  78%; 0.0 mph; N/A deg; 0.0 mph gust;  43.2F; 0.00" rain  ... skipped.
2019-05-21 00:02:00 EDT (1558411320); 29.379";  49.6F;  78%; 0.0 mph; N/A deg; 0.0 mph gust;  43.1F; 0.00" rain  ... skipped.
2019-05-21 22:42:00 EDT (1558492920); 29.406";  55.4F;  67%; 1.3 mph;  68 deg; 0.0 mph gust;  44.6F; 0.00" rain  ... skipped.
pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-22 --test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-22
Number of archive records:     1440
Number of WU records:          288
Number of missing records:     3

Missing records:
2019-05-22 00:00:00 EDT (1558497600); 29.406";  54.0F;  65%; 2.0 mph;  76 deg; 3.6 mph gust;  42.5F; 0.00" rain  ... skipped.
2019-05-22 00:01:00 EDT (1558497660); 29.406";  54.0F;  65%; 2.3 mph;  64 deg; 2.5 mph gust;  42.5F; 0.00" rain  ... skipped.
2019-05-22 00:02:00 EDT (1558497720); 29.406";  54.0F;  65%; 2.5 mph;  61 deg; 2.9 mph gust;  42.5F; 0.00" rain  ... skipped.
pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-23 --test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-23
Number of archive records:     1438
Number of WU records:          288
Number of missing records:     4

Missing records:
2019-05-23 00:00:00 EDT (1558584000); 29.244";  59.9F;  90%; 1.3 mph; 158 deg; 1.8 mph gust;  56.9F; 0.00" rain  ... skipped.
2019-05-23 00:01:00 EDT (1558584060); 29.244";  59.8F;  90%; 1.3 mph; 159 deg; 1.8 mph gust;  56.8F; 0.00" rain  ... skipped.
2019-05-23 00:02:00 EDT (1558584120); 29.244";  59.7F;  90%; 1.3 mph; 160 deg; 1.3 mph gust;  56.8F; 0.00" rain  ... skipped.
2019-05-23 14:52:00 EDT (1558637520); 29.161";  83.8F;  46%; 3.6 mph; 248 deg;12.3 mph gust;  60.8F; 0.00" rain  ... skipped.


Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-24
Number of archive records:     1436
Number of WU records:          260
Number of missing records:     149

Missing records:
2019-05-24 00:00:00 EDT (1558670400); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 00:01:00 EDT (1558670460); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 00:02:00 EDT (1558670520); 29.320";  63.0F;  70%; 0.0 mph; N/A deg; 0.0 mph gust;  52.9F; 0.00" rain  ... skipped.
2019-05-24 18:22:00 EDT (1558736520); 29.359";  67.5F;  68%; 2.5 mph;  99 deg; 5.1 mph gust;  56.5F; 0.00" rain  ... skipped.
2019-05-24 18:40:00 EDT (1558737600); 29.353";  67.3F;  68%; 2.0 mph; 118 deg; 4.0 mph gust;  56.3F; 0.00" rain  ... skipped.
2019-05-24 18:41:00 EDT (1558737660); 29.353";  67.3F;  68%; 2.0 mph; 118 deg; 3.6 mph gust;  56.3F; 0.00" rain  ... skipped.
...

When forming the query URL, I suspect I need to convert the local date to the UTC date, since I was within 4 hours of UTC midnight.  The docs will probably confirm it.

The thing is, late last night (US/Eastern), the number was over 1300.
If it's the UTC vs. EDT date in the URL, I would expect the records to line up when querying what is querying what is now yesterday.

And for today, so far, seems reasonable.

pi@nixie:/usr/share/weewx4/bin $ ./wunderfixer --epsilon=125 --date=2019-05-25 --test --verbose | more
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-25
Number of archive records:     526
Number of WU records:          106
Number of missing records:     3

Missing records:
2019-05-25 00:00:00 EDT (1558756800); 29.329";  61.4F;  75%; 0.0 mph; N/A deg; 0.0 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 00:01:00 EDT (1558756860); 29.329";  61.3F;  75%; 0.0 mph; N/A deg; 3.1 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 00:02:00 EDT (1558756920); 29.329";  61.3F;  75%; 0.0 mph; N/A deg; 2.5 mph gust;  53.3F; 0.00" rain  ... skipped.


Once I figure out what really happened with yesterday's data, late last night, and
If I manage to find some long-lost brain cells over the weekend, I'd still like to "normalize" the local records to 5-minute boundaries, and not bother to flag the in-between ones for re-upload, since WU seems to drop them on purpose.

There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, because WU is only going to keep 00:00:00, and then later it will keep whatever is closest to 00:05:00.   That's been the behavior of wunderfixer vs. WU for as long as I've been working with it, however.  :-/

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
May 26, 2019, 5:43:23 PM5/26/19
to weewx...@googlegroups.com
Tested on 4.x (python 2 and 3) and 3.9.x. (python 2).

Ultimately went with apikey instead of apiKey in weewx.conf, which was to be more consistent with other parameter names.

If you are willing to pass "--apikey=0123456789012345678901" on wunderfixer command-line, then you only need wunderfixer.

Unrelated to these changes, FYI you probably also want --epsilon=125 because of silly WU "rounding" errors on fractional minutes.

If you wish to put "apikey=0123456789012345678901" in the WU section of weewx.conf, then you also need restx.py.

API Key is generated in the "Member Settings" section of wunderground.com ("My Profile" > "Member Settings" > "API Keys").

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

gjr80

unread,
May 26, 2019, 10:59:56 PM5/26/19
to weewx-user
On Saturday, 25 May 2019 22:54:25 UTC+10, Leon Shaner wrote:

There is no point re-uploading 00:00:00, then 00:01:00, then 00:02:00, because WU is only going to keep 00:00:00, and then later it will keep whatever is closest to 00:05:00.   That's been the behavior of wunderfixer vs. WU for as long as I've been working with it, however.  :-/

Maybe, maybe not. The problem is WU acts as a black box that accepts your data at (more or less) whatever interval you choose WU then does who knows what with the data and (usually) 5 minute interval records appear. You could argue that re-uploading of data should be done in the same manner as it was originally uploaded.

Gary

Leon Shaner

unread,
May 26, 2019, 11:16:53 PM5/26/19
to weewx...@googlegroups.com
Gary,

In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have seen where if wunderfixer is used persistently enough (say more than 10 times spread out every 20 minutes across a total span of 200 minutes) then it will keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even want to spend cycles chasing what's really going on there.  I would rather just only have wunderfixer ignore most records except those nearest the 5-minute boundaries that WU seems to most care about.

I have just now solved the "align to 5-minute boundaries" exercise with:

_gen_rows =  dbmanager.genSql("""SELECT dateTime FROM archive WHERE dateTime>=? AND dateTime<? GROUP BY (dateTime - 60 ) / 300 """,
                                  (start_ts, end_ts))

Example:

$ /usr/share/weewx4/bin/wunderfixer_403 --epsilon=125 --date=2019-05-25 --test --verbose | head -30
Using configuration file /usr/share/weewx4/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-25
Number of archive records:     289
Number of WU records:          260
Number of missing records:     288

Missing records:
2019-05-25 00:05:00 EDT (1558757100); 29.326";  61.3F;  76%; 0.0 mph; N/A deg; 3.6 mph gust;  53.7F; 0.00" rain  ... skipped.
2019-05-25 00:10:00 EDT (1558757400); 29.326";  61.2F;  76%; 0.0 mph; N/A deg; 2.0 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 00:15:00 EDT (1558757700); 29.326";  61.2F;  76%; 1.3 mph;  93 deg; 3.1 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 00:20:00 EDT (1558758000); 29.317";  61.1F;  76%; 1.3 mph;  93 deg; 2.5 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 00:25:00 EDT (1558758300); 29.317";  61.0F;  77%; 1.3 mph; 121 deg; 4.3 mph gust;  53.7F; 0.00" rain  ... skipped.
2019-05-25 00:30:00 EDT (1558758600); 29.317";  61.0F;  76%; 1.3 mph; 121 deg; 3.6 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 00:35:00 EDT (1558758900); 29.309";  61.0F;  77%; 1.8 mph; 112 deg; 4.0 mph gust;  53.7F; 0.00" rain  ... skipped.
2019-05-25 00:40:00 EDT (1558759200); 29.309";  60.9F;  76%; 1.8 mph; 112 deg; 3.1 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 00:45:00 EDT (1558759500); 29.309";  60.7F;  77%; 1.8 mph; 116 deg; 5.8 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 00:50:00 EDT (1558759800); 29.306";  60.6F;  77%; 1.8 mph; 116 deg; 5.1 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 00:55:00 EDT (1558760100); 29.306";  60.6F;  77%; 2.5 mph; 113 deg; 2.9 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 01:00:00 EDT (1558760400); 29.306";  60.5F;  77%; 2.5 mph; 113 deg; 2.5 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 01:05:00 EDT (1558760700); 29.303";  60.5F;  77%; 1.3 mph; 120 deg; 3.6 mph gust;  53.2F; 0.00" rain  ... skipped.
2019-05-25 01:10:00 EDT (1558761000); 29.303";  60.4F;  78%; 1.3 mph; 120 deg; 1.3 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 01:15:00 EDT (1558761300); 29.303";  60.4F;  78%; 1.3 mph; 110 deg; 4.7 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 01:20:00 EDT (1558761600); 29.285";  60.4F;  78%; 1.3 mph; 110 deg; 5.1 mph gust;  53.5F; 0.00" rain  ... skipped.
2019-05-25 01:25:00 EDT (1558761900); 29.285";  60.3F;  78%; 2.0 mph; 102 deg; 3.1 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 01:30:00 EDT (1558762200); 29.285";  60.3F;  78%; 2.0 mph; 102 deg; 5.4 mph gust;  53.4F; 0.00" rain  ... skipped.
2019-05-25 01:35:00 EDT (1558762500); 29.282";  60.2F;  78%; 2.5 mph; 123 deg; 2.5 mph gust;  53.3F; 0.00" rain  ... skipped.
2019-05-25 01:40:00 EDT (1558762800); 29.282";  60.1F;  78%; 2.5 mph; 123 deg; 3.6 mph gust;  53.2F; 0.00" rain  ... skipped.
2019-05-25 01:45:00 EDT (1558763100); 29.282";  60.1F;  78%; 2.0 mph; 111 deg; 5.4 mph gust;  53.2F; 0.00" rain  ... skipped.

But as you can see, I still have to solve the REAL problem that something is in disagreement about the localtime vs. UTC-time w.r.t. the dates in question.

I'm now forced 100% on that part, which is the real issue.

(Being that if I am within X hours of UTC where X = my localtime UTC offset, I get 100% records being marked as missing).

The problem is likely in that same block of code I pasted above, but I don't yet have my head around it.  :-/  Could also be a bug in the IBM API w.r.t. date specification in the URL a la 20190525, which I glean is supposed to be normalized ON THIER END to the local time of the PWS (since they know where it's located)...but could be a bug in their API implementation and isn't being handled properly.   :-/

I think I recently mentioned, that I am a persistent SOB, so it won't be long before I put the nail in this coffin in a way that is in our favor.  LOL  =D

If you can see any shorter paths to a more reliable outcome than I have achieved so far, then you know know know I will be very grateful.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

gjr80

unread,
May 27, 2019, 12:12:54 AM5/27/19
to weewx-user
On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
Gary,

In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have seen where if wunderfixer is used persistently enough (say more than 10 times spread out every 20 minutes across a total span of 200 minutes) then it will keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even want to spend cycles chasing what's really going on there.  I would rather just only have wunderfixer ignore most records except those nearest the 5-minute boundaries that WU seems to most care about.

I am not suggesting we/you/anyone should try to solve WU issues/problems/whatever, merely making the point that WU is a blackbox, nobody seems to really know what it does with the data that it is fed. If we are going to have WU recreate/create 'missing' data then the logical approach would be to feed it with the data it was orignally fed rather than trying to guess what it wants.
I am not sure what local/UTC issue you refer to. When I do a api.weather.com/v2/pws/history query on a station to the east of Greenwich I am returned all records for the date specified (eg 20190525 gives me all records for 25 May 2019), each record contains an epoch timestamp which is correct and consistent with 25 May 2019. Everything is as I would expect. However, when I perform the same query on a station to the west of Greenwich I am returned records for the day before the date specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each record contains an epoch timestamp but the timestamp is for the previous day Ie 24 May 2019. I have checked a number of data records in the stations history table and WU is definitely returning the midnight to midnight data for the day before. I have confirmed this behaviour with a number of stations both east and west of Greenwich.

I don't think there is a local/UTC time issue, I think WU is having some implementation issues and for stations west of Greenwich they are returning the wrong day of data.

Gary

Leon Shaner

unread,
May 27, 2019, 12:28:58 AM5/27/19
to weewx...@googlegroups.com
Gary,

It's late, so I'll respond to the rest later, but...

The problem here is that if we compare our local every 1-minute records to WU's query that only shows every 5-minute records, then we'll keep re-uploading the the "missing" 1-minute records every time wunderfixer is called.  This amounts to pointless hits against WU's infrastructure, since pragmatically it's very clear they almost always drop all records not closely aligned to 5-minute "buckets."

I'm trying to get to the heart of the matter re: what wunderfixer was really designed to do, which is to make sure what we have locally is consistent with what WU will actually REPORT, and when there are gaps in what WU *should* report, and we have local data to fill in those gaps, then re-upload.  BUT to not gratuitously re-upload data that WU generally throws away (they just don't seem to care about storing data at < 5-minute intervals).


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

On May 27, 2019, at 12:12 AM, gjr80 <gjrod...@gmail.com> wrote:

On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
Gary,

In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have seen where if wunderfixer is used persistently enough (say more than 10 times spread out every 20 minutes across a total span of 200 minutes) then it will keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even want to spend cycles chasing what's really going on there.  I would rather just only have wunderfixer ignore most records except those nearest the 5-minute boundaries that WU seems to most care about.

Gary,

In practice, WU seems to discard data that is not close to their APPARENTly preferred 5-minute "normalization buckets."

I upload via rapidfire *and* regular loop on 1-minute intervals, and irregardless of same, the queries only ever show the records most closely aligned to their "preferred" 5-minute "buckets."

I would love to see them preserve the same interval that I upload, but either by design or omission or bug, they simply don't.  :-(
Could be the layers in-between are dropping data by code-exception.
Could be it's deliberate.
I'm not here to debug their code unless they want to start paying me a worthy salary. ;-)

Well...  MOST of the time the behavior I have described seems to be true.  :-/

While WU normally SEEMS to only keeps records on 5-minute boundaries, I have seen where if wunderfixer is used persistently enough (say more than 10 times spread out every 20 minutes across a total span of 200 minutes) then it will keep records that are more frequent than every 5-minutes.

Right now / especially lately, things are so sporadic with WU that I don't even want to spend cycles chasing what's really going on there.  I would rather just only have wunderfixer ignore most records except those nearest the 5-minute boundaries that WU seems to most care about.

Leon Shaner

unread,
May 27, 2019, 7:35:25 AM5/27/19
to weewx...@googlegroups.com

On May 27, 2019, at 12:12 AM, gjr80 <gjrod...@gmail.com> wrote:

On Monday, 27 May 2019 13:16:53 UTC+10, Leon Shaner wrote:
[snip]
If you can see any shorter paths to a more reliable outcome than I have achieved so far, then you know know know I will be very grateful.  =D

I am not sure what local/UTC issue you refer to. When I do a api.weather.com/v2/pws/history query on a station to the east of Greenwich I am returned all records for the date specified (eg 20190525 gives me all records for 25 May 2019), each record contains an epoch timestamp which is correct and consistent with 25 May 2019. Everything is as I would expect. However, when I perform the same query on a station to the west of Greenwich I am returned records for the day before the date specified (ie 20190525 gives me all records for 24 May 2019 not 25 May 2019), again each record contains an epoch timestamp but the timestamp is for the previous day Ie 24 May 2019. I have checked a number of data records in the stations history table and WU is definitely returning the midnight to midnight data for the day before. I have confirmed this behaviour with a number of stations both east and west of Greenwich.

I don't think there is a local/UTC time issue, I think WU is having some implementation issues and for stations west of Greenwich they are returning the wrong day of data.

Thanks, Gary!  This was all very helpful.
In addition to what you've described across the east vs west of GMT, I get similar behavior if I am within X hours of my local UTC offset when querying my own station.
Last night as soon as localtime rolled over midnight, the queries for the previous day were correct.

--Leon


Gary

Rod Yager

unread,
May 29, 2019, 6:14:07 PM5/29/19
to weewx-user
There is definitely a time zone issue. I am in the Sydney Australia timezone (UTC +10 hours).

It is currently 8am local time on May 30, 2019.   (10pm May 29, 2019 UTC)

If I execute

./wunderfixer --verbose --date=2019-05-29 --epsilon=125


I get


Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 'archive_mysql'

Weather Underground Station:   xxxxxxx

Date to check:                 2019-05-29

Number of archive records:     288

Number of WU records:          97

Number of missing records:     288


Now WU actually has 288 records for 2019-05-29.

But it only has 97 records for 2019-05-30.


So it is clear that wunderfixer is downloading the record data for 2019-05-30 from WeatherUnderground and trying to match them with the local records for 2019-30-29.

Of course, they all mismatch, and so wunderfixer concludes that it must upload all the data for 2019-05-29.


Hope this narrows down the search for a solution.


Rod

Leon Shaner

unread,
May 29, 2019, 7:29:16 PM5/29/19
to weewx...@googlegroups.com
Hi, Rod,

Yes and thanks for adding yet another confirmation of the issue.  =D

I can show that if I do the query within X hours of my offset of UTC, what actually happens is they report 288 records from the day PRIOR to the one I am asking about.
For example, I ask for 20190528 and they give me records for 20190527, so *that* is why wunderfixer "thinks" it needs to re-upload everything.

I am in contact with IBM about it and have shown them irrefutable proof of the issue.
They didn't respond back yet, which I expect is because the proof was irrefutable.  Ha!  ;-)

I expect that they're investigating and would rather respond from a position of understanding, or with any luck maybe even a quick fix.  =D

I meant to follow-up with IBM again this morning, but got waylaid, so I'll do that now.
Thanks again, and for the reminder.  =D 


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Rod Yager

unread,
May 29, 2019, 8:53:27 PM5/29/19
to weewx-user
Further to this, it has now rolled past 10am here, so the local date is now the same as the UTC date. (ie. local time May 30 2019 10:40 AM = May 30 2019 00:40 AM UTC).

Now I get:

./wunderfixer --verbose --date=2019-05-29 --epsilon=125

Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 'archive_mysql'

Weather Underground Station:   xxxxx

Date to check:                 2019-05-29

Number of archive records:     288

Number of WU records:          288

Number of missing records:     0

[root@moses bin]# ./wunderfixer --verbose --date=2019-05-30 --epsilon=125

Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 'archive_mysql'

Weather Underground Station:   xxxxxx

Date to check:                 2019-05-30

Number of archive records:     128

Number of WU records:          127

Number of missing records:     1


This means that WU is now actually providing the records for the date requested, rather than the day after the requested date. 
So it seems that what is happening is that WU is determining whether or not the current date at the station location is the same as the UTC date.
If it is, it returns the data for the date as in the request. But if the local date is different, it makes an (unwanted) adjustment for the date difference.

Rod 
To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

Leon Shaner

unread,
May 30, 2019, 12:41:29 AM5/30/19
to weewx...@googlegroups.com
Rod,
Thanks again for this.

Since the in-progress version of wunderfixer doesn't really show you the dates that come back from WU, I have written a tool just to debug the dates.

The command-line input and basing of defaults on weewx.conf works the same as wunderfixer, except it doesn't look at your DB at all.  It only prints out datestamps in various incarnations coming back from WU and as compared to your system's localtime.

It would be helpful to see the "wunderdates" output at times like you've shown below, a la before and after your localtime rolls around to UTC midnight.

Since you are at UTC + 10, another interesting time would be 11+ hours on either side of UTC midnight, in addition to within 9 or less hours.  This is just to make sure we're covering all the corner cases.

Gary reported a difference for stations that are east vs. west of GMT, and I expect we're really chasing the same bug in that there is some bad math WU is doing based on UTC offset, but since an offset can be +/-, the effects go in opposite directions date-wise, depending on which side of the UTC dateline your station is located.

At least that is what I surmise may be happening, but the wunderdates utility should shed light one way or the other.  =D

The wunderdates utility is available at the links below.

Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output.

What I've been doing is saving the output to files for use with sdiff (side-by-side) diff.
Or you can just compare the head and tail of each file individually.

Example for my system before and after my local time on 5-28 once it was at/after 8 p.m. here, which is within 4 hours of UTC midnight (I am at UTC-4).

This output is optimized for screen widths 203 or wider.  Sorry.  :S
Mainly the last two data fields in the output tell what we need to know.


pi@nixie:/var/tmp $ head -n 3 59:19:28:5_wu.txt  0:20:28:5_wu.txt
==> 59:19:28:5_wu.txt <==
Using configuration file /usr/share/weewx4/weewx.conf.
epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59
epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59

==> 0:20:28:5_wu.txt <==
Using configuration file /usr/share/weewx4/weewx.conf.
epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 utcepoch: 1558930299 date_epoch_utc: 2019-05-27 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59
epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 utcepoch: 1558930599 date_epoch_utc: 2019-05-27 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59


pi@nixie:/var/tmp $ tail -n 3 59:19:28:5_wu.txt  0:20:28:5_wu.txt
==> 59:19:28:5_wu.txt <==
epoch: 1559087699 date_epoch_local: 2019-05-28 19:54:59 utcepoch: 1559088099 date_epoch_utc: 2019-05-28 23:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T23:54:59Z obsTimeLocal: 2019-05-28 19:54:59
epoch: 1559087702 date_epoch_local: 2019-05-28 19:55:02 utcepoch: 1559088102 date_epoch_utc: 2019-05-28 23:55:02 tz: America/New_York obsTimeUtc: 2019-05-28T23:55:02Z obsTimeLocal: 2019-05-28 19:55:02
Number of WU records:          240

==> 0:20:28:5_wu.txt <==
epoch: 1559015699 date_epoch_local: 2019-05-27 23:54:59 utcepoch: 1559016099 date_epoch_utc: 2019-05-28 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:54:59Z obsTimeLocal: 2019-05-27 23:54:59
epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 utcepoch: 1559016399 date_epoch_utc: 2019-05-28 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59
Number of WU records:          288


But after my system rolled over midnight localtime, results returned to the correct dates when asking for 5-28:

pi@nixie:/var/tmp $ head -3 0:0:29:5_wu.txt
Using configuration file /usr/share/weewx4/weewx.conf.
epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 utcepoch: 1559016699 date_epoch_utc: 2019-05-28 04:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59
epoch: 1559016599 date_epoch_local: 2019-05-28 00:09:59 utcepoch: 1559016999 date_epoch_utc: 2019-05-28 04:09:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:09:59Z obsTimeLocal: 2019-05-28 00:09:59

pi@nixie:/var/tmp $ tail -3 0:0:29:5_wu.txt
epoch: 1559102099 date_epoch_local: 2019-05-28 23:54:59 utcepoch: 1559102499 date_epoch_utc: 2019-05-29 03:54:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:54:59Z obsTimeLocal: 2019-05-28 23:54:59
epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 utcepoch: 1559102799 date_epoch_utc: 2019-05-29 03:59:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59
Number of WU records:          288
Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3570f548-caf3-43e8-858d-1b4d8c87d84a%40googlegroups.com.

Rod Yager

unread,
May 30, 2019, 6:16:27 AM5/30/19
to weewx-user

Have downloaded the wunderdates. 

I am in the UTC+10 timezone.

So far, my experience is that when the current time is between 00:00+10 and 09:59+10  (14:00UTC to 23:59 UTC) (so the local date is one day in advance of the UTC date), the API fetches data for the local-time day immediately following the date specified in the URL.  ie a request with date=20190528 returns the data observations recorded on 20190529 local time.

At all other times (so far) the api returns the data for the local time day specified in the request URL (as expected).

From what I understand, you are in the UTC-4 time zone and your experience is that when the current time is between 20:00-4 and 23:59-4 (00:00UTC to 03:59UTC) (so the local date is one day behind the UTC date), the API fetches data for the local-time date immediately before the date specified in the URL. ie a request with date=20190529 returns the data for observations recorded on 20190527 local time. 

At all other times, I gather the api returns the data for the local time day specified in the request URL (as expected).

I'll conduct further tests, and will certainly report anything that doesn't fit my hypothesis that the unwanted behaviour reflects the difference between the current local time date and the current UTC date. But its difficult to see how there could be more to it than this. (Famous last words).

It's now 20:14+10 on 20190530  (10:14UTC on 20190503) The last three lines of  produced by ./wunderdates  are (as expected)

epoch: 1559210400 date_epoch_local: 2019-05-30 20:00:00 utcepoch: 1559209400 date_epoch_utc: 2019-05-30 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T10:00:00Z obsTimeLocal: 2019-05-30 20:00:00

epoch: 1559210700 date_epoch_local: 2019-05-30 20:05:00 utcepoch: 1559209700 date_epoch_utc: 2019-05-30 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T10:05:00Z obsTimeLocal: 2019-05-30 20:05:00

Number of WU records:          242



Rod

Rod Yager

unread,
May 30, 2019, 10:48:09 PM5/30/19
to weewx-user

I constructed a cron job which exectued  wunderdates --date=2019-05-30  twice every hour, once 2 minutes before the hour and once 2 minutes after the hour.

Consistent with my earlier post, between 00:00+10 and 09:59+10 the reply from the api had data which belonged to 2019-05-31 - the day after the requested date. Before midnight, and after 10am local time, the reply from the api had data which belonged to 2019-05-30 (as requested).

I've pasted below some representative samples of the last 3 lines of the wunderdates output.

Before midnight on May 30: Request is for May 30 data. Replies are May 30 data  Local Date = UTC date.

Thu May 30 20:58:02 2019 UTC+10
epoch: 1559213100 date_epoch_local: 2019-05-30 20:45:00 utcepoch: 1559212100 date_epoch_utc: 2019-05-30 10:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T10:45:00Z obsTimeLocal: 2019-05-30 20:45:00
epoch: 1559213400 date_epoch_local: 2019-05-30 20:50:00 utcepoch: 1559212400 date_epoch_utc: 2019-05-30 10:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T10:50:00Z obsTimeLocal: 2019-05-30 20:50:00
Number of WU records:          251


Thu May 30 22:58:03 2019 UTC+10
epoch: 1559220000 date_epoch_local: 2019-05-30 22:40:00 utcepoch: 1559219000 date_epoch_utc: 2019-05-30 12:40:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T12:40:00Z obsTimeLocal: 2019-05-30 22:40:00
epoch: 1559220300 date_epoch_local: 2019-05-30 22:45:00 utcepoch: 1559219300 date_epoch_utc: 2019-05-30 12:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T12:45:00Z obsTimeLocal: 2019-05-30 22:45:00
epoch: 1559220600 date_epoch_local: 2019-05-30 22:50:00 utcepoch: 1559219600 date_epoch_utc: 2019-05-30 12:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T12:50:00Z obsTimeLocal: 2019-05-30 22:50:00
Number of WU records:          275


Thu May 30 23:58:02 2019 UTC+10
epoch: 1559223900 date_epoch_local: 2019-05-30 23:45:00 utcepoch: 1559222900 date_epoch_utc: 2019-05-30 13:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:45:00Z obsTimeLocal: 2019-05-30 23:45:00
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
Number of WU records:          287


10 hour period after midnight on May 31. Request is for May 30 data. Replies are May 31 data.  Local Date is 1 day ahead of UTC date

Fri May 31 00:02:02 2019 UTC+10
Could not get Weather Underground data.
Reason: No JSON object could be decoded
Could not get Weather Underground data. Exiting. 


Fri May 31 00:58:02 2019  UTC+10
epoch: 1559227500 date_epoch_local: 2019-05-31 00:45:00 utcepoch: 1559226500 date_epoch_utc: 2019-05-30 14:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:45:00Z obsTimeLocal: 2019-05-31 00:45:00
epoch: 1559227800 date_epoch_local: 2019-05-31 00:50:00 utcepoch: 1559226800 date_epoch_utc: 2019-05-30 14:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:50:00Z obsTimeLocal: 2019-05-31 00:50:00
Number of WU records:          11


Fri May 31 02:58:03 2019 UTC+10
epoch: 1559234700 date_epoch_local: 2019-05-31 02:45:00 utcepoch: 1559233700 date_epoch_utc: 2019-05-30 16:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T16:45:00Z obsTimeLocal: 2019-05-31 02:45:00
epoch: 1559235000 date_epoch_local: 2019-05-31 02:50:00 utcepoch: 1559234000 date_epoch_utc: 2019-05-30 16:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T16:50:00Z obsTimeLocal: 2019-05-31 02:50:00
Number of WU records:          35


Fri May 31 05:58:02 2019 UTC+10
epoch: 1559245500 date_epoch_local: 2019-05-31 05:45:00 utcepoch: 1559244500 date_epoch_utc: 2019-05-30 19:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T19:45:00Z obsTimeLocal: 2019-05-31 05:45:00
epoch: 1559245800 date_epoch_local: 2019-05-31 05:50:00 utcepoch: 1559244800 date_epoch_utc: 2019-05-30 19:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T19:50:00Z obsTimeLocal: 2019-05-31 05:50:00
Number of WU records:          71


Fri May 31 08:58:03 2019 UTC+10
epoch: 1559256300 date_epoch_local: 2019-05-31 08:45:00 utcepoch: 1559255300 date_epoch_utc: 2019-05-30 22:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T22:45:00Z obsTimeLocal: 2019-05-31 08:45:00
epoch: 1559256600 date_epoch_local: 2019-05-31 08:50:00 utcepoch: 1559255600 date_epoch_utc: 2019-05-30 22:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T22:50:00Z obsTimeLocal: 2019-05-31 08:50:00
Number of WU records:          107


Fri May 31 09:58:03 2019 UTC+10
epoch: 1559259900 date_epoch_local: 2019-05-31 09:45:00 utcepoch: 1559258900 date_epoch_utc: 2019-05-30 23:45:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T23:45:00Z obsTimeLocal: 2019-05-31 09:45:00
epoch: 1559260200 date_epoch_local: 2019-05-31 09:50:00 utcepoch: 1559259200 date_epoch_utc: 2019-05-30 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T23:50:00Z obsTimeLocal: 2019-05-31 09:50:00
Number of WU records:          119

After 10AM on May 31: Request is for May 30 data. Replies are May 30 data  Local Date = UTC date.

Fri May 31 10:02:02 2019 UTC+10
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
epoch: 1559224500 date_epoch_local: 2019-05-30 23:55:00 utcepoch: 1559223500 date_epoch_utc: 2019-05-30 13:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:55:00Z obsTimeLocal: 2019-05-30 23:55:00
Number of WU records:          288

Fri May 31 11:02:02 2019 UTC+10
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
epoch: 1559224500 date_epoch_local: 2019-05-30 23:55:00 utcepoch: 1559223500 date_epoch_utc: 2019-05-30 13:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:55:00Z obsTimeLocal: 2019-05-30 23:55:00
Number of WU records:          288

Fri May 31 12:02:03 2019 UTC+10
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 utcepoch: 1559223200 date_epoch_utc: 2019-05-30 13:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
epoch: 1559224500 date_epoch_local: 2019-05-30 23:55:00 utcepoch: 1559223500 date_epoch_utc: 2019-05-30 13:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:55:00Z obsTimeLocal: 2019-05-30 23:55:00
Number of WU records:          288


I think this is enough to describe definitively what happens in UTC+X timezones.

Rod




Andrew Milner

unread,
May 31, 2019, 1:14:28 AM5/31/19
to weewx-user
..... so does this just simplify to the fact that in the api the date is taken as being the UTC date?
ie if one requests data for 20190529, and timezone is +3 you get back data for UTC00:00 29 may to 23:59 29 may 
which is 03:00 29 may local to 23:59 29 may local AND 00:00 30 may local - 02:59 30 may local

so to get the data for 29 may local it would be necessary to 
1. get data for utc 28 may and discard utc 00:00 - 20:59
2. get data for utc 29 may and discard utc 21:00 - 23:59

or have i not understood correctly?

Rod Yager

unread,
May 31, 2019, 1:50:48 AM5/31/19
to weewx...@googlegroups.com
No.  You always get all available data for a complete day (00:00:00 to 23:59:59 local time).

So in UTC+10, the first entry in the returned data is   00:00:00 UTC+10  (14:00:00 UTC the previous day)  and the last entry in the returned data is 23:59:59 UTC+10  (13:59:59 UTC  same day)
and in UTC-4, the first entry in the returned data is   00:00:00 UTC-4  (04:00:00 UTC same day)  and the last entry in the returned data is 23:59:59 UTC-4  (03:59:59 UTC the  next day)

Consider a request for data for 20190520 made in the UTC+10 timezone.
If the request is made at 01:00 on May 31 local time,   (15:00 May 30 UTC), the API will return all data for 20190521. ie. Observations made from 00:00 May 21 to 23:59 May 21 local time.  (14:00 May 20 UTC to 13:59 May 21 UTC)
If the request is made at 11:00 on May 31 local time,   (01:00 May 31 UTC), the API will return all data for 20190520. ie. Observations made from 00:00 May 20 to 23:59 May 20 local time. (14:00 May 19 UTC to 13:59 May 20 UTC)


I think that the same request for data for 20190520 made in the UTC-4 timezone has the following behaviour (judging from what I’ve seen reported on this thread)
If the request is made at 11:00 on May 30 local time,   (15:00 May 31 UTC), the API will return all data for 20190520. ie. Observations made from 00:00 May 20 to 23:59 May 20 local time. (04:00 May 20 UTC to 03:59 May 21 UTC)
If the request is made at 21:00 on May 30 local time,   (01:00 May 31 UTC), the API will return all data for 20190519. ie. Observations made from 00:00 May 19 to 23:59 May 19 local time (04:00 May 19 UTC to 03:59 May 20 UTC)


So it appears that Weather Underground is doing the following.

1. Determining whether current date at the station is the same as the current date UTC.
2. If the two dates are different, adjust the date in the request to “allow for” the date difference.
3. Return the data between 00:00 and 23:59 (local time) on the adjusted date. 

Step 2 is unwanted and undesirable.


If Weather Underground doesn’t fix this, wunderfixer needs to work out whether the local time date is the same as the UTC date. If it is, send the request as usual.
If the local time date is currently ahead of the UTC date, adjust the request to ask for the day before the date you actually want.
If the local time date is currently behind the UTC date, adjust the request to ask for the day after the date you actually want.


Rod







--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Leon Shaner

unread,
May 31, 2019, 9:46:35 AM5/31/19
to weewx...@googlegroups.com
Andrew,

We can't use this as a workaround, but what you're suggesting is essentially what WU should be doing with the date that we provide in the query URL.
WU should be doing the conversion to station localtime and returning the values for that date, relative to the station.
But they seem to be returning the values for the date relative to UTC instead.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
Jun 1, 2019, 1:18:01 PM6/1/19
to weewx...@googlegroups.com
All,

I'm testing a new approach.  Below you will find links to an updated wunderdates utility that can be used to validate whether I am on the right track.
The wunderdates utility simply dumps out timestamp related records from what WU returns from the query.

We're looking mainly at the obsTimeUtc and obsTimeLocal values, which were demonstrated under the prior approach to be returning the wrong dates when within the stations localtime +/- offset from UTC.

The new approach is to detect if the requested date is 'today' and if so, use a different API URL that already assumes 'today' and will hopefully not be subject to the UTC offset bugs we've been chasing with the historical data API URL.

I have my crontab set to do another test approaching my UTC offset, just after coming within the offset, and then again just before and after midnight localtime.
(Same test I did before, but now with the new approach in place).

Here is the utility for anyone else that wants to check out the behavior:


Which version you pull will depend on which base weewx you are running.
Pull the one that matches your weewx version and place it in bin, next to wunderfixer, and it will take the same arguments as wunderfixer.

You can just try wunderfixer as you normally would (with --apikey) and then run wunderdates(3 or 4) with exact same arguments to be able to see what actual timestamps WU is sending back for the date queried.
Parameters like --epsilon don't have any effect in the case of wunderdates, but I left it there so you don't have to change options when running the util to get the debug output.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 1, 2019, 6:03:38 PM6/1/19
to weewx...@googlegroups.com
Dear Leon,

Your approach is flawed. It won’t work for historical data.

If you run

wunderdates —date=2019-01-01   at 9pm your time   you will receive the data for 2018-12-31  (because your station is a day behind UTC at the time of the request, so it gives you the data for the previous day).

On the other hand, if you run 

wunderdates —date=2019-01-01   at 9am your time   you will receive the data for 2019-01-01  (because your station is in the same day as UTC at the time of the request).

If I do the same thing (I’m in UTC+10)

wunderdates —date=2019-01-01   at 9pm my time  produces the data for 2019-01-01 (because my station is in the same day as UTC at the time of the request)  but
wunderdates —date=2019-01-01   at 9am my time produces the data for 2019-01-02 (because my station is a day ahead of UTC at the time of the request) 

Rod

Leon Shaner

unread,
Jun 1, 2019, 7:27:58 PM6/1/19
to weewx...@googlegroups.com
Rod,

Thanks for testing.  Well, WU's approach is flawed, not mine, really.  :S

It means that both API's, the one I just added for 'today" and the one from before that was for 'historical' (including today), both have the same bug.

For now, the best workaround I can think of is to only ever run the new wunderfixer only ever for the previous day, only ever after UTC has rolled over midnight.   :-(
E.g. when it is 2019-06-02 in UTC, you can get the right results with wunderfixer if you ask for 2019-06-01 only.    In other words, always delay your wunderfixer fix ups by a day, operating always on yesterday's date (relative to UTC) after UTC has rolled over to the next day.

Scroll to see the output of me running wunderdates on my local machine in UTC-4 against your station in UTC+10.

Because I am asking for 2019-06-01, which is also my local date, the logic will use the URL that queries for 'today' and instead of returning obsTimeLocal records, it is returning obsTimeUtc records for the requested date.  We can see that it's returning 2019-06-02 relative to your station, even though I asked for 2019-06-01.

Sure, I could go a bit further and use a related API query to check the timezone of your station and use date conversions to check if the date requested is 'today' relative to the station requested, but really this is intended to be run on the same machine as weewx, so the localtime date should match that of the station.  It means there isn't any real reason for me to workaround their bug.  In fact even if I did, then the logic would just fall through to the historical data URL, which we already know is buggy.

For now, I'll just report this to IBM a bug in the 'today' API, similar to the bug in the 'historical' API.

$ ./wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | more

Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-01
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 utcepoch: 1559398000 date_epoch_utc: 2019-06-01 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 utcepoch: 1559398300 date_epoch_utc: 2019-06-01 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
...


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 1, 2019, 11:08:56 PM6/1/19
to weewx...@googlegroups.com
Dear Leon,

If WU doesn’t fix their bug, the solution is really quite simple.

We need to make 3 requests to WU for the data: one request for the date we are interested in, one request for the day before and one request for the day after. 

eg. If we are intending to upload missing data for 2019-01-01 we need to ask WU for the records it has for 2018-12-31, 2019-01-01 and 2019-01-02. 

One of these will have the information pertaining to the day we are interested in.  

We then need to filter the three responses from WU, only keeping the responses that pertain to the day we are interested in.

At this point, we have exactly what we want from WU and so we compare that with the local database, and upload the data that is missing in the WU record, just as we always did.


in your timezone (UTC-4) if you ask for data for your station the correct information will be in 
  • the same day request:  if the request is made between 00:00 and 19:59   (at those times, the UTC date is the same as the date where you are )
  • the request for the next day: if the request is made between 20:00 and 23:59  (at those times, the UTC date is one day ahead the date where you are)

If you request information for my station (I’ve sent you the details in a PM), the correct information will be in
  • the same day request: if the request is made between 10:00 and 19:59 your time  (at those times,  the UTC date is the same as the date where my station is located)
  • the request for the previous day: if the request is made between 0:00 and 09:59 or between 20:00 and 23:59 your time (at those times, the UTC date is one day behind the date where my station is located)

Rod



Leon Shaner

unread,
Jun 2, 2019, 10:21:02 AM6/2/19
to weewx...@googlegroups.com
Rod,

It's interesting that the "workaround" that I've already done, is actually working 100% for me.   I got good results when combining the 'today' API when asking for the current date, and only using the 'historical' result when asking for dates other than the current date -- relative to station local time.

Here is my KMIDEARB5 station's transition from 7:59 p.m. to 8:00 p.m, where time moves to within the UTC-4 offset:

$ diff 59:19:1:6_wu.txt 0:20:1:6_wu.txt
241c241,242
< Number of WU records:          239
---
> epoch: 1559433599 date_epoch_local: 2019-06-01 19:59:59 utcepoch: 1559433999 date_epoch_utc: 2019-06-01 23:59:59 tz: America/New_York obsTimeUtc: 2019-06-01T23:59:59Z obsTimeLocal: 2019-06-01 19:59:59
> Number of WU records:          240

The above is exactly correct -- the second query is one minute later and is after/on the 5-minute record "boundary" hence one additional record.

And then as my station transition from 11:59 p.m. to midnight next day.  Before midnight the 'today' API is used, but after midnight the 'historical' API will be used, which was always working for me after midnight (station local time):

$ tail -n 4 59:23:1:6_wu.txt 0:0:2:6_wu.txt
==> 59:23:1:6_wu.txt <==
epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59
epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59
epoch: 1559447701 date_epoch_local: 2019-06-01 23:55:01 utcepoch: 1559448101 date_epoch_utc: 2019-06-02 03:55:01 tz: America/New_York obsTimeUtc: 2019-06-02T03:55:01Z obsTimeLocal: 2019-06-01 23:55:01
Number of WU records:          288

==> 0:0:2:6_wu.txt <==
epoch: 1559447399 date_epoch_local: 2019-06-01 23:49:59 utcepoch: 1559447799 date_epoch_utc: 2019-06-02 03:49:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:49:59Z obsTimeLocal: 2019-06-01 23:49:59
epoch: 1559447699 date_epoch_local: 2019-06-01 23:54:59 utcepoch: 1559448099 date_epoch_utc: 2019-06-02 03:54:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:54:59Z obsTimeLocal: 2019-06-01 23:54:59
epoch: 1559447999 date_epoch_local: 2019-06-01 23:59:59 utcepoch: 1559448399 date_epoch_utc: 2019-06-02 03:59:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:59:59Z obsTimeLocal: 2019-06-01 23:59:59
Number of WU records:          288

There again, the second query is a minute later, so one more result, but with explanation...  ;-)

There is a slight difference between the 'today' and 'historical' result.
The first result, using the 'today' API ran at 1 minute before midnight, which was on the cusp of posting the 1-minute record nearest actual midnight.
The 'today' result shows two time-stamps very close to 11:55 p.m.
Where-as the 'historical' result replaces the later ~11:55 p.m. result with the ~midnight result.

Those differences are "in the noise" and inconsequential.  Even the default wunderfixer --epsilon=120 will filter out such a duplicate.

I am happy that this new approach with the combined 'today' and 'historical' queries is at least effective for stations west of the UTC dateline.
It means I have easily worked around half of the WU API bug.

Now to further press IBM to fix the other half of the bug...  =D
(Even while I continue to think about ways to *cleanly* workaround the bug for stations east of UTC dateline).

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
Jun 2, 2019, 3:47:23 PM6/2/19
to weewx...@googlegroups.com
Rod,

I've spent quite a bit of time working on a "cleaner" workaround for the WU bugs.
But just when I thought I had landed on the "reasonable" approach, I hit a snag.

As a reminder, with the latest approach, all is fine for stations west of the UTC date line.   What I am working on now is a workaround for stations east of the UTC date line, where the data returned is a day after the date requested.

The wall that I just hit is that the WU behavior around this latest bug changes, when looking historically.  :-(

From today's date going back to 2019-05-28, the query returns obsTimeLocal data from the day after the date requested.  This is the bug we are trying to workaround.
To compensate, I can ask for the prior date to get the date we really want.

HOWEVER, starting with 2019-05-26 (and prior), the query returns obsTimeLocal data for the date that was actually requested.

This leads to the inability to get data for 2019-05-27, because the actual query for that date returns 2019-05-28 datum, and the compensation date of 2019-05-26 returns actual data for 2019-05-26.

I think I am done trying to work around this bug.  IBM needs to own it.

See here -- scroll to second to last example where 2019-05-27 is requested, but when I tried 2019-05-26, I got back data for the actual 2019-05-26, so compensation did not work.  :-(

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-02 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-02
WU API obsTimeLocal:           2019-06-03
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-06-01
WU API obsTimeLocal:           2019-06-02
epoch: 1559397600 date_epoch_local: 2019-06-01 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-01 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-01 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-01
WU API obsTimeLocal:           2019-06-02
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-31
WU API obsTimeLocal:           2019-06-01
epoch: 1559311200 date_epoch_local: 2019-05-31 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 00:00:00
epoch: 1559311500 date_epoch_local: 2019-05-31 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 00:05:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-31 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-31
WU API obsTimeLocal:           2019-06-01
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-30
WU API obsTimeLocal:           2019-05-31
epoch: 1559224800 date_epoch_local: 2019-05-30 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 00:00:00
epoch: 1559225100 date_epoch_local: 2019-05-30 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 00:05:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-30 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-30
WU API obsTimeLocal:           2019-05-31
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-29
WU API obsTimeLocal:           2019-05-30
epoch: 1559138400 date_epoch_local: 2019-05-29 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 00:00:00
epoch: 1559138700 date_epoch_local: 2019-05-29 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 00:05:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-29 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-29
WU API obsTimeLocal:           2019-05-30
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-28
WU API obsTimeLocal:           2019-05-29
epoch: 1559052000 date_epoch_local: 2019-05-28 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00
epoch: 1559052300 date_epoch_local: 2019-05-28 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-28 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-28
WU API obsTimeLocal:           2019-05-29
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-27
WU API obsTimeLocal:           2019-05-28
epoch: 1558965600 date_epoch_local: 2019-05-27 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
epoch: 1558965900 date_epoch_local: 2019-05-27 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
Number of WU records:          288

/usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-27
WU API obsTimeLocal:           2019-05-28
WU API WRONG DATE !!!!!!!!!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-26
WU API obsTimeLocal:           2019-05-26
WU API COMPSENSATION FAILURE! ABORTING!!!
Number of WU records:          0

No results returned from Weather Underground (perhaps a bad station name??).


pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-26
WU API obsTimeLocal:           2019-05-26
epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
epoch: 1558793400 date_epoch_local: 2019-05-25 10:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00
epoch: 1558793700 date_epoch_local: 2019-05-25 10:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00
epoch: 1558794000 date_epoch_local: 2019-05-25 10:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00
Number of WU records:          288
Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 2, 2019, 5:51:12 PM6/2/19
to weewx...@googlegroups.com
Well, it turns out that the East of the UTC date line bug only occurs for the latest week. (I’d assumed that testing involving requests for the last few days would show the behaviour, but that turned out to be wrong.)

At the moment, I’m a day ahead of UTC.  Here is the output for requests for 2019-05-27 (within the last week and giving data for the 2019-05-28) and for 2019-05-26 (not within the last week and giving the correct data for 2019-05-06)  [I repeated the 2019-05-27 responses to show that they hadn’t happened coincidentally at a time where WU fixed the problem.]  All dates I’ve tested prior to 2019-05-27 also work as we would like. All dates within the last week give data for the day following the day in the request.

[rodyager@moses ~]$ date
Mon Jun  3 07:32:55 AEST 2019
[rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-27
epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: 1558964600 date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
epoch: 1558965900 date_epoch_local: 2019-05-28 00:05:00 utcepoch: 1558964900 date_epoch_utc: 2019-05-27 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
epoch: 1558966200 date_epoch_local: 2019-05-28 00:10:00 utcepoch: 1558965200 date_epoch_utc: 2019-05-27 14:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:10:00Z obsTimeLocal: 2019-05-28 00:10:00
epoch: 1558966500 date_epoch_local: 2019-05-28 00:15:00 utcepoch: 1558965500 date_epoch_utc: 2019-05-27 14:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:15:00Z obsTimeLocal: 2019-05-28 00:15:00
epoch: 1558966800 date_epoch_local: 2019-05-28 00:20:00 utcepoch: 1558965800 date_epoch_utc: 2019-05-27 14:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:20:00Z obsTimeLocal: 2019-05-28 00:20:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-26
epoch: 1558792800 date_epoch_local: 2019-05-26 00:00:00 utcepoch: 1558791800 date_epoch_utc: 2019-05-25 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
epoch: 1558793100 date_epoch_local: 2019-05-26 00:05:00 utcepoch: 1558792100 date_epoch_utc: 2019-05-25 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
epoch: 1558793400 date_epoch_local: 2019-05-26 00:10:00 utcepoch: 1558792400 date_epoch_utc: 2019-05-25 14:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:10:00Z obsTimeLocal: 2019-05-26 00:10:00
epoch: 1558793700 date_epoch_local: 2019-05-26 00:15:00 utcepoch: 1558792700 date_epoch_utc: 2019-05-25 14:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:15:00Z obsTimeLocal: 2019-05-26 00:15:00
epoch: 1558794000 date_epoch_local: 2019-05-26 00:20:00 utcepoch: 1558793000 date_epoch_utc: 2019-05-25 14:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:20:00Z obsTimeLocal: 2019-05-26 00:20:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates --epsilon=125 --date 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 8 ; tail -n 1)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-27
epoch: 1558965600 date_epoch_local: 2019-05-28 00:00:00 utcepoch: 1558964600 date_epoch_utc: 2019-05-27 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
epoch: 1558965900 date_epoch_local: 2019-05-28 00:05:00 utcepoch: 1558964900 date_epoch_utc: 2019-05-27 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
epoch: 1558966200 date_epoch_local: 2019-05-28 00:10:00 utcepoch: 1558965200 date_epoch_utc: 2019-05-27 14:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:10:00Z obsTimeLocal: 2019-05-28 00:10:00
epoch: 1558966500 date_epoch_local: 2019-05-28 00:15:00 utcepoch: 1558965500 date_epoch_utc: 2019-05-27 14:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:15:00Z obsTimeLocal: 2019-05-28 00:15:00
epoch: 1558966800 date_epoch_local: 2019-05-28 00:20:00 utcepoch: 1558965800 date_epoch_utc: 2019-05-27 14:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:20:00Z obsTimeLocal: 2019-05-28 00:20:00
Number of WU records:          288


Rod


Leon Shaner

unread,
Jun 2, 2019, 7:12:27 PM6/2/19
to weewx...@googlegroups.com
Rod,

Thanks again for the added validation of what I saw earlier.
Never minding for now the issues with last week's data, what I am now most concerned to know is whether there is any point throughout the PRESENT day (your local time) that the latest wunderdates is showing the wrong results?

Recall that for the present day I am using a 'today' API which is for the 'today' results.
That method is working for my station at all times during the PRESENT day (even within my UTC offset of local midnight).

I thought I understood you to say that those changes were no good for you at certain times of the PRESENT day, such as before 10 a.m. your local time, was it?

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 2, 2019, 7:29:01 PM6/2/19
to weewx...@googlegroups.com
The “today" result works as desired when I am ahead of the UTC date (as I am at the moment and will be for the next 33 minutes)

[root@moses bin]# date
Mon Jun  3 09:25:10 AEST 2019
[root@moses bin]# /home/weewx/bin/wunderdates3 --epsilon=125 --station ISYDNEY155 --test --verbose | (head -n 9 ; tail -n 1)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-03
epoch: 1559484000 date_epoch_local: 2019-06-03 00:00:00 utcepoch: 1559483000 date_epoch_utc: 2019-06-02 14:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00
epoch: 1559484300 date_epoch_local: 2019-06-03 00:05:00 utcepoch: 1559483300 date_epoch_utc: 2019-06-02 14:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00
epoch: 1559484600 date_epoch_local: 2019-06-03 00:10:00 utcepoch: 1559483600 date_epoch_utc: 2019-06-02 14:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00
epoch: 1559484900 date_epoch_local: 2019-06-03 00:15:00 utcepoch: 1559483900 date_epoch_utc: 2019-06-02 14:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00
epoch: 1559485200 date_epoch_local: 2019-06-03 00:20:00 utcepoch: 1559484200 date_epoch_utc: 2019-06-02 14:20:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:20:00Z obsTimeLocal: 2019-06-03 00:20:00
epoch: 1559485500 date_epoch_local: 2019-06-03 00:25:00 utcepoch: 1559484500 date_epoch_utc: 2019-06-02 14:25:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:25:00Z obsTimeLocal: 2019-06-03 00:25:00
Number of WU records:          112


So the problem only exists for 6 days [ yesterday and the preceeding days]

Rod

Leon Shaner

unread,
Jun 2, 2019, 7:34:00 PM6/2/19
to weewx...@googlegroups.com
Rod,

Thanks!  I checked your station via 'today' API just after I sent that note and it looks fine to me, also.

I've set a timer for just before the top of the hour to capture your station's results, and again just after.  Should be the same as you doing the test from wunderdates.  LOL  =D

I'mma bug IBM tomorrow as much as my schedule allows.  I want them to schedule a real-time meeting so we can go over the outstanding issues and chip away at fixing them in priority order.   Clearly e-mail to one person doesn't cut it as a "support" model, even if that one person were, "all that and a bag of chips."  :-/

I may have mentioned that I am a persistent SOB.  LOL  ;-)
We'll get there.  =D

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Leon Shaner

unread,
Jun 2, 2019, 8:42:11 PM6/2/19
to weewx...@googlegroups.com
Rod,

And finally.  The 'today' queries against your station check out before and after 10 a.m. your local time (before and after UTC midnight).

So I think we have a viable workaround for dealing with WeeWX outages in real-time.
But unfortunately, we are dependent upon IBM to fix the issue with dates going back to 2019-05-27 showing the day after the date requested.

$ cat /var/tmp/ISYDNEY155_59\:19\:3\:6_wu.txt | (head -9 ; tail -1)
Using configuration file /usr/share/weewx4/weewx.conf.
WU API obsTimeLocal:           2019-06-03
epoch: 1559484000 date_epoch_local: 2019-06-02 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00
epoch: 1559484300 date_epoch_local: 2019-06-02 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00
epoch: 1559484600 date_epoch_local: 2019-06-02 10:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00
epoch: 1559484900 date_epoch_local: 2019-06-02 10:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00
epoch: 1559485200 date_epoch_local: 2019-06-02 10:20:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:20:00Z obsTimeLocal: 2019-06-03 00:20:00
epoch: 1559485500 date_epoch_local: 2019-06-02 10:25:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:25:00Z obsTimeLocal: 2019-06-03 00:25:00
epoch: 1559485800 date_epoch_local: 2019-06-02 10:30:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:30:00Z obsTimeLocal: 2019-06-03 00:30:00
Number of WU records:          119

$ cat /var/tmp/ISYDNEY155_0\:20\:3\:6_wu.txt | (head -9 ; tail -1)
Using configuration file /usr/share/weewx4/weewx.conf.
WU API obsTimeLocal:           2019-06-03
epoch: 1559484000 date_epoch_local: 2019-06-02 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00
epoch: 1559484300 date_epoch_local: 2019-06-02 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00
epoch: 1559484600 date_epoch_local: 2019-06-02 10:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00
epoch: 1559484900 date_epoch_local: 2019-06-02 10:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00
epoch: 1559485200 date_epoch_local: 2019-06-02 10:20:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:20:00Z obsTimeLocal: 2019-06-03 00:20:00
epoch: 1559485500 date_epoch_local: 2019-06-02 10:25:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:25:00Z obsTimeLocal: 2019-06-03 00:25:00
epoch: 1559485800 date_epoch_local: 2019-06-02 10:30:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:30:00Z obsTimeLocal: 2019-06-03 00:30:00
Number of WU records:          119


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 2, 2019, 10:04:28 PM6/2/19
to weewx...@googlegroups.com
Agreed. 

Rod

Sent from my iPhone

Leon Shaner

unread,
Jun 2, 2019, 11:11:46 PM6/2/19
to weewx...@googlegroups.com
Rod,

Here's the crazy thing.  Once UTC local time passed into 2019-06-03, all the 'history' API queries for your station started coming back with the correct date, from 2019-06-02 back through the odd date, 2019-05-27, and prior.

And on the opposite end, now that UTC already is on tomorrow's date (relative to my local time), I am having to compensate in the opposite direction for 'history' queries, except on 2019-05-27 or prior (older dates than that are fine).

This is sometimes the nature of chasing bugs.  There's no explaining them without more information about what is happening on the other side of the query.  :-/

Relevant queries for my station:

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-02 --station KMIDEARB5 --test --verbose | (head -n 8 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-06-02
WU API obsTimeLocal:           2019-06-02
epoch: 1559448299 date_epoch_local: 2019-06-02 00:04:59 tz: America/New_York obsTimeUtc: 2019-06-02T04:04:59Z obsTimeLocal: 2019-06-02 00:04:59
epoch: 1559448599 date_epoch_local: 2019-06-02 00:09:59 tz: America/New_York obsTimeUtc: 2019-06-02T04:09:59Z obsTimeLocal: 2019-06-02 00:09:59
epoch: 1559448899 date_epoch_local: 2019-06-02 00:14:59 tz: America/New_York obsTimeUtc: 2019-06-02T04:14:59Z obsTimeLocal: 2019-06-02 00:14:59
epoch: 1559449199 date_epoch_local: 2019-06-02 00:19:59 tz: America/New_York obsTimeUtc: 2019-06-02T04:19:59Z obsTimeLocal: 2019-06-02 00:19:59
epoch: 1559530496 date_epoch_local: 2019-06-02 22:54:56 tz: America/New_York obsTimeUtc: 2019-06-03T02:54:56Z obsTimeLocal: 2019-06-02 22:54:56
Number of WU records:          275

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-06-01 --station KMIDEARB5 --test --verbose | (head -n 8 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-06-01
WU API obsTimeLocal:           2019-05-31
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-06-02
WU API obsTimeLocal:           2019-06-01
epoch: 1559361899 date_epoch_local: 2019-06-01 00:04:59 tz: America/New_York obsTimeUtc: 2019-06-01T04:04:59Z obsTimeLocal: 2019-06-01 00:04:59
epoch: 1559447999 date_epoch_local: 2019-06-01 23:59:59 tz: America/New_York obsTimeUtc: 2019-06-02T03:59:59Z obsTimeLocal: 2019-06-01 23:59:59
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-28 --station KMIDEARB5 --test --verbose | (head -n 8 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-28
WU API obsTimeLocal:           2019-05-27
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-29
WU API obsTimeLocal:           2019-05-28
epoch: 1559016299 date_epoch_local: 2019-05-28 00:04:59 tz: America/New_York obsTimeUtc: 2019-05-28T04:04:59Z obsTimeLocal: 2019-05-28 00:04:59
epoch: 1559102399 date_epoch_local: 2019-05-28 23:59:59 tz: America/New_York obsTimeUtc: 2019-05-29T03:59:59Z obsTimeLocal: 2019-05-28 23:59:59
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 --station KMIDEARB5 --test --verbose | (head -n 8 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   KMIDEARB5
Date to check:                 2019-05-27
WU API obsTimeLocal:           2019-05-27
epoch: 1558929899 date_epoch_local: 2019-05-27 00:04:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:04:59Z obsTimeLocal: 2019-05-27 00:04:59
epoch: 1558930199 date_epoch_local: 2019-05-27 00:09:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:09:59Z obsTimeLocal: 2019-05-27 00:09:59
epoch: 1558930497 date_epoch_local: 2019-05-27 00:14:57 tz: America/New_York obsTimeUtc: 2019-05-27T04:14:57Z obsTimeLocal: 2019-05-27 00:14:57
epoch: 1558930799 date_epoch_local: 2019-05-27 00:19:59 tz: America/New_York obsTimeUtc: 2019-05-27T04:19:59Z obsTimeLocal: 2019-05-27 00:19:59
epoch: 1559015999 date_epoch_local: 2019-05-27 23:59:59 tz: America/New_York obsTimeUtc: 2019-05-28T03:59:59Z obsTimeLocal: 2019-05-27 23:59:59
Number of WU records:          288


--- transition ---

Relevant queries for your station (compensation logic never has to kick in):

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 6 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-26
WU API obsTimeLocal:           2019-05-26
epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
epoch: 1558878900 date_epoch_local: 2019-05-26 09:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T13:55:00Z obsTimeLocal: 2019-05-26 23:55:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-28 --station ISYDNEY155 --test --verbose | (head -n 6 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-28
WU API obsTimeLocal:           2019-05-28
epoch: 1558965600 date_epoch_local: 2019-05-27 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:00:00Z obsTimeLocal: 2019-05-28 00:00:00
epoch: 1558965900 date_epoch_local: 2019-05-27 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T14:05:00Z obsTimeLocal: 2019-05-28 00:05:00
epoch: 1559051700 date_epoch_local: 2019-05-28 09:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T13:55:00Z obsTimeLocal: 2019-05-28 23:55:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-27 --station ISYDNEY155 --test --verbose | (head -n 6 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-27
WU API obsTimeLocal:           2019-05-27
epoch: 1558879200 date_epoch_local: 2019-05-26 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
epoch: 1558879500 date_epoch_local: 2019-05-26 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
epoch: 1558965300 date_epoch_local: 2019-05-27 09:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-26 --station ISYDNEY155 --test --verbose | (head -n 6 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-26
WU API obsTimeLocal:           2019-05-26
epoch: 1558792800 date_epoch_local: 2019-05-25 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:00:00Z obsTimeLocal: 2019-05-26 00:00:00
epoch: 1558793100 date_epoch_local: 2019-05-25 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T14:05:00Z obsTimeLocal: 2019-05-26 00:05:00
epoch: 1558878900 date_epoch_local: 2019-05-26 09:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T13:55:00Z obsTimeLocal: 2019-05-26 23:55:00
Number of WU records:          288

pi@nixie:/usr/share/weewx4/bin $ /usr/share/weewx4/bin/wunderdates --epsilon=125 --date 2019-05-25 --station ISYDNEY155 --test --verbose | (head -n 6 ; tail -n 2)
Using configuration file /usr/share/weewx4/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-25
WU API obsTimeLocal:           2019-05-25
epoch: 1558706400 date_epoch_local: 2019-05-24 10:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-24T14:00:00Z obsTimeLocal: 2019-05-25 00:00:00
epoch: 1558706700 date_epoch_local: 2019-05-24 10:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-24T14:05:00Z obsTimeLocal: 2019-05-25 00:05:00
epoch: 1558792500 date_epoch_local: 2019-05-25 09:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-25T13:55:00Z obsTimeLocal: 2019-05-25 23:55:00
Number of WU records:          288


I'll check tomorrow to see if the 'history' API queries against your station revert again to bad behavior after midnight your local time.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 3, 2019, 1:33:11 AM6/3/19
to weewx...@googlegroups.com
This is what I was always trying to explain to you.

To summarise what we now know:

Requests for data for days which are more than one week in the past always return the correct data.

Requests for data for days in the last week returns the correct data when the CURRENT date in the timezone in which station is located is the same as the CURRENT date in the UTC+0 time zone.

Requests for data for days in the last week returns the data for the day after the date in the request if the CURRENT date in the timezone in which the station is located is ahead of the CURRENT date in the UTC+0 timezone
(happens for stations located east of the Greenwich meridian)

Requests for data for days in the last week returns the data for the day before the date in the request if the CURRENT date in the timezone in which the station is located is behind of the CURRENT date in the UTC+0 timezone
(happens for stations located west of the Greenwich meridian)

I can understand why WU might mistakenly think that they need to adjust for “date differences”. The thing I can’t understand from the WU perspective is why they have a different logic to the processing of requests for recent data to that of more distant data? 

Rod

Leon Shaner

unread,
Jun 3, 2019, 3:35:05 PM6/3/19
to weewx...@googlegroups.com
All,
There are updates to the wunderdates utilities, to implement the following:

Add +/- 1-day compensation logic to workaround WU 'historical' API bug, depending on whether station is east/west of UTC, and whether it is within X hours of UTC midnight, relative to the local station time.

These wunderdates utilities are for debug purposes only to aid in testing the WU behavior, while IBM works to fix the actual bug (no intention of adding this logic to wunderfixer, which will remain broken until IBM fixes their API).

The bugs are well known / well documented.
The continued use of wunderdates is just to keep an eye on IBM's progress in fixing the bugs, e.g. keep an eye out for if the behavior changes / improves.

Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 3, 2019, 5:39:25 PM6/3/19
to weewx...@googlegroups.com
Dear Leon,

Thanks for all your work on this. This works as well as we are going to be able to manage unless and until WU fixes the bug. 

The remaining issue affects just one date, the  “change-over” date when WU transitions to returning the correct data - and will only bite for stations east of UTC, and only in the hours that they are ahead of UTC.

Currently, for me, that date is 2019-05-28. When it originally asks for 2019-05-28, it returns the data for 2019-05-29. Wunderfixer then tries to compensate by asking for 2019-05-27. But 2019-05-27 is a date before the WU bug, and so it returns the data for 2019-05-27. We can’t work around this, because no request to WU will actually return the data we want. The only fix is to wait a few hours until the UTC date is aligned with the Sydney date.

Good news is that for stations west of UTC, this won’t happen as the bug causes a duplicate day, not a missing day.

Here’s the output for the requests for my station from 2019-05-27 (before the WU bug kicks in) to today 2019-06-04.


[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-27  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-27
WU API obsTimeLocal:           2019-05-27
epoch: 1558879200 date_epoch_local: 2019-05-27 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:00:00Z obsTimeLocal: 2019-05-27 00:00:00
epoch: 1558879500 date_epoch_local: 2019-05-27 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:05:00Z obsTimeLocal: 2019-05-27 00:05:00
epoch: 1558879800 date_epoch_local: 2019-05-27 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:10:00Z obsTimeLocal: 2019-05-27 00:10:00
epoch: 1558880100 date_epoch_local: 2019-05-27 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:15:00Z obsTimeLocal: 2019-05-27 00:15:00
epoch: 1558880400 date_epoch_local: 2019-05-27 00:20:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:20:00Z obsTimeLocal: 2019-05-27 00:20:00
epoch: 1558880700 date_epoch_local: 2019-05-27 00:25:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:25:00Z obsTimeLocal: 2019-05-27 00:25:00
epoch: 1558881000 date_epoch_local: 2019-05-27 00:30:00 tz: Australia/Sydney obsTimeUtc: 2019-05-26T14:30:00Z obsTimeLocal: 2019-05-27 00:30:00
epoch: 1558965000 date_epoch_local: 2019-05-27 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T13:50:00Z obsTimeLocal: 2019-05-27 23:50:00
epoch: 1558965300 date_epoch_local: 2019-05-27 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-27T13:55:00Z obsTimeLocal: 2019-05-27 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-28  --test --verbose | (head -n 11 ; tail -n 3)

No results returned from Weather Underground (perhaps a bad station name??).
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-28
WU API obsTimeLocal:           2019-05-29
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-27
WU API obsTimeLocal:           2019-05-27
WU API COMPENSATION FAILURE! ABORTING!!!
Number of WU records:          0
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-29  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-29
WU API obsTimeLocal:           2019-05-30
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-28
WU API obsTimeLocal:           2019-05-29
epoch: 1559052000 date_epoch_local: 2019-05-29 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:00:00Z obsTimeLocal: 2019-05-29 00:00:00
epoch: 1559052300 date_epoch_local: 2019-05-29 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:05:00Z obsTimeLocal: 2019-05-29 00:05:00
epoch: 1559052600 date_epoch_local: 2019-05-29 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:10:00Z obsTimeLocal: 2019-05-29 00:10:00
epoch: 1559052900 date_epoch_local: 2019-05-29 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-28T14:15:00Z obsTimeLocal: 2019-05-29 00:15:00
epoch: 1559137800 date_epoch_local: 2019-05-29 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T13:50:00Z obsTimeLocal: 2019-05-29 23:50:00
epoch: 1559138100 date_epoch_local: 2019-05-29 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T13:55:00Z obsTimeLocal: 2019-05-29 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-30  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-30
WU API obsTimeLocal:           2019-05-31
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-29
WU API obsTimeLocal:           2019-05-30
epoch: 1559138400 date_epoch_local: 2019-05-30 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:00:00Z obsTimeLocal: 2019-05-30 00:00:00
epoch: 1559138700 date_epoch_local: 2019-05-30 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:05:00Z obsTimeLocal: 2019-05-30 00:05:00
epoch: 1559139000 date_epoch_local: 2019-05-30 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:10:00Z obsTimeLocal: 2019-05-30 00:10:00
epoch: 1559139300 date_epoch_local: 2019-05-30 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-29T14:15:00Z obsTimeLocal: 2019-05-30 00:15:00
epoch: 1559224200 date_epoch_local: 2019-05-30 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:50:00Z obsTimeLocal: 2019-05-30 23:50:00
epoch: 1559224500 date_epoch_local: 2019-05-30 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T13:55:00Z obsTimeLocal: 2019-05-30 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-05-31  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-05-31
WU API obsTimeLocal:           2019-06-01
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-30
WU API obsTimeLocal:           2019-05-31
epoch: 1559224800 date_epoch_local: 2019-05-31 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:00:00Z obsTimeLocal: 2019-05-31 00:00:00
epoch: 1559225100 date_epoch_local: 2019-05-31 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:05:00Z obsTimeLocal: 2019-05-31 00:05:00
epoch: 1559225400 date_epoch_local: 2019-05-31 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:10:00Z obsTimeLocal: 2019-05-31 00:10:00
epoch: 1559225700 date_epoch_local: 2019-05-31 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-30T14:15:00Z obsTimeLocal: 2019-05-31 00:15:00
epoch: 1559310600 date_epoch_local: 2019-05-31 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T13:50:00Z obsTimeLocal: 2019-05-31 23:50:00
epoch: 1559310900 date_epoch_local: 2019-05-31 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T13:55:00Z obsTimeLocal: 2019-05-31 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-01  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-01
WU API obsTimeLocal:           2019-06-02
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-05-31
WU API obsTimeLocal:           2019-06-01
epoch: 1559311200 date_epoch_local: 2019-06-01 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:00:00Z obsTimeLocal: 2019-06-01 00:00:00
epoch: 1559311500 date_epoch_local: 2019-06-01 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:05:00Z obsTimeLocal: 2019-06-01 00:05:00
epoch: 1559311800 date_epoch_local: 2019-06-01 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:10:00Z obsTimeLocal: 2019-06-01 00:10:00
epoch: 1559312100 date_epoch_local: 2019-06-01 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-05-31T14:15:00Z obsTimeLocal: 2019-06-01 00:15:00
epoch: 1559397000 date_epoch_local: 2019-06-01 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T13:50:00Z obsTimeLocal: 2019-06-01 23:50:00
epoch: 1559397300 date_epoch_local: 2019-06-01 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T13:55:00Z obsTimeLocal: 2019-06-01 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-02  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-02
WU API obsTimeLocal:           2019-06-03
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-06-01
WU API obsTimeLocal:           2019-06-02
epoch: 1559397600 date_epoch_local: 2019-06-02 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:00:00Z obsTimeLocal: 2019-06-02 00:00:00
epoch: 1559397900 date_epoch_local: 2019-06-02 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:05:00Z obsTimeLocal: 2019-06-02 00:05:00
epoch: 1559398200 date_epoch_local: 2019-06-02 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:10:00Z obsTimeLocal: 2019-06-02 00:10:00
epoch: 1559398500 date_epoch_local: 2019-06-02 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-01T14:15:00Z obsTimeLocal: 2019-06-02 00:15:00
epoch: 1559483400 date_epoch_local: 2019-06-02 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T13:50:00Z obsTimeLocal: 2019-06-02 23:50:00
epoch: 1559483700 date_epoch_local: 2019-06-02 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T13:55:00Z obsTimeLocal: 2019-06-02 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125 --date 2019-06-03  --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-03
WU API obsTimeLocal:           2019-06-04
WU API RETURNED WRONG DATE!!!!!!!!!!!!!!!
WU API COMPENSATION DATE:      2019-06-02
WU API obsTimeLocal:           2019-06-03
epoch: 1559484000 date_epoch_local: 2019-06-03 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:00:00Z obsTimeLocal: 2019-06-03 00:00:00
epoch: 1559484300 date_epoch_local: 2019-06-03 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:05:00Z obsTimeLocal: 2019-06-03 00:05:00
epoch: 1559484600 date_epoch_local: 2019-06-03 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:10:00Z obsTimeLocal: 2019-06-03 00:10:00
epoch: 1559484900 date_epoch_local: 2019-06-03 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-02T14:15:00Z obsTimeLocal: 2019-06-03 00:15:00
epoch: 1559569800 date_epoch_local: 2019-06-03 23:50:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T13:50:00Z obsTimeLocal: 2019-06-03 23:50:00
epoch: 1559570100 date_epoch_local: 2019-06-03 23:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T13:55:00Z obsTimeLocal: 2019-06-03 23:55:00
Number of WU records:          288
[rodyager@moses ~]$ /home/weewx/bin/wunderdates3 --epsilon=125   --test --verbose | (head -n 11 ; tail -n 3)
Using configuration file /home/weewx/weewx.conf.
Weather Underground Station:   ISYDNEY155
Date to check:                 2019-06-04
WU API obsTimeLocal:           2019-06-04
epoch: 1559570400 date_epoch_local: 2019-06-04 00:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:00:00Z obsTimeLocal: 2019-06-04 00:00:00
epoch: 1559570700 date_epoch_local: 2019-06-04 00:05:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:05:00Z obsTimeLocal: 2019-06-04 00:05:00
epoch: 1559571000 date_epoch_local: 2019-06-04 00:10:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:10:00Z obsTimeLocal: 2019-06-04 00:10:00
epoch: 1559571300 date_epoch_local: 2019-06-04 00:15:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:15:00Z obsTimeLocal: 2019-06-04 00:15:00
epoch: 1559571600 date_epoch_local: 2019-06-04 00:20:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:20:00Z obsTimeLocal: 2019-06-04 00:20:00
epoch: 1559571900 date_epoch_local: 2019-06-04 00:25:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:25:00Z obsTimeLocal: 2019-06-04 00:25:00
epoch: 1559572200 date_epoch_local: 2019-06-04 00:30:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T14:30:00Z obsTimeLocal: 2019-06-04 00:30:00
epoch: 1559595300 date_epoch_local: 2019-06-04 06:55:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T20:55:00Z obsTimeLocal: 2019-06-04 06:55:00
epoch: 1559595600 date_epoch_local: 2019-06-04 07:00:00 tz: Australia/Sydney obsTimeUtc: 2019-06-03T21:00:00Z obsTimeLocal: 2019-06-04 07:00:00
Number of WU records:          85

Regards,

Rod

Leon Shaner

unread,
Jun 3, 2019, 5:55:43 PM6/3/19
to weewx...@googlegroups.com
Thanks, Rod!  =D

Those are precisely the same tests I ran and exact same results that noted, before  before publishing.  =D
Plus a ton of checks against my own station, KMIDEARB5, west of UTC.

As before, even more important than these 'historical' API tests is to query against the current day at different times throughout the day, such as before and after your midnight and before and after UTC midnight.  That logic which uses the 'today' API (rapid) for today is the one we will likely keep no matter what IBM does with the well-known bugs in the 'historical' API.

(I can't test the 'today' logic 100% without changing my localtime on my system and I don't want to do that, which is why I appreciate help from others to validate the code).  =D


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad Pro)

Rod Yager

unread,
Jun 3, 2019, 6:21:54 PM6/3/19
to weewx...@googlegroups.com
I’ve already migrated the today logic into my local version of wunderfixer, which runs as a cron job at 05:58, 11:58, 17:58 and 23:58.

This works as expected here.

The only circumstance I can see in which it would break is if I make a request for today’s data  at 23:59:59.95 on my machine. The transmission delay (and possibly differences in the clocks) will mean that WU receives my request at a time when it thinks it is 00:00:00.01 on the following day (my time) and so it will return the (empty) data for the day after the date on which I made the request. We can’t worry about such things.


Rod



Thomas Keffer

unread,
Jul 18, 2019, 11:11:44 AM7/18/19
to weewx-user

I've ported wunderfixer to the new WU API. Commit 32c35ce on the development branch.


Unfortunately, it looks like the WU no longer allows re-posting old records, so the utility may no longer be useful. I'd be interested in other people's experience.


Also, as I understand it, there are issues for people living east of Greenwich. I live west, so I wasn't able to check that. Other people's experiences would be welcome.


NB: this is on the development branch. You will have to clone the weewx GitHub repository, then check out the development branch. 


-tk

Leon Shaner

unread,
Jul 18, 2019, 3:21:58 PM7/18/19
to Thomas Keffer, weewx-user
Tom,

There's nothing wrong with the old methods of reposting old records to WU.

IBM is pretty clueless about the old interfaces we use from the pre-IBM WU days.  When IBM says they don't yet support re-uploading old records via API, they are most surely talking about the new API's.

I can attest the methods that wunderfixer uses to re-upload do still work, but they are subject to the old caveats that I mentioned in a thread some while back.  WU quite clearly "normalizes" records to 5-minute boundaries even if weewx is using rapidfire or an upload interval shorter than 5-minutes.

SO you can keep trying to re-upload records at n-minute or n-second boundaries < 5-minutes, but they won't "stick."
 Furthermore, even re-uploading of records that are aligned to 5-minute boundaries seemed to take several re-uploads before they stick, but in actuality it's just that they can take a very long time to "post" full-circle such that they become available on a subsequent query (that was the case even with the old query interface, and is still the case with the new API for querying "today" and the other API for querying past days).

 I have found that if I just spread out the calls to wunderfixer enough, then the re-uploads do stick (after waiting long enough) and I don't need to waste cycles re-uploading a second, third, or n-th time.

Below is a recent example after a USB hang issue on my RPI.  You can see how frequently my watchdog script runs, and what it found each time and that in fact it was successful in re-uploading the missing records.
The only exception seems to be the records very close to midnight tend to usually not stick because WU is strangely keeping a record at 11:59:59.59 instead of accepting the 00:00:00.00  record.

NOTE:  That in my case I have altered the weewx db query to pre-align the local records to 5-minute boundaries (even though my archive interval is every 1-minute). Pre-filtering the weewx archive query to 5-minute boundaries cuts out a huge number of "false-positive" missing records that wouldn't "stick" on the WU side even if I did let wunderfixer try to upload them.   NOTE2:  Those every 1-minute records didn't "stick" even before IBM went about breaking things wholesale across the board.

Example (wunderfixer with --espsilon 125 --timeout 30, and with archive queries normalized to 5-minute boundaries):

Wed 17 Jul 01:17:01 EDT 2019 WeeWX: Wunderfixer timestamp: (1563340621)
Wed 17 Jul 01:17:01 EDT 2019 Using configuration file /usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:17:01 EDT 2019 Date to check:                 2019-07-16
Wed 17 Jul 01:17:01 EDT 2019 Number of archive records:     275
Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:          273
Wed 17 Jul 01:17:01 EDT 2019 Number of missing records:     3
Wed 17 Jul 01:17:01 EDT 2019
Wed 17 Jul 01:17:01 EDT 2019 Missing records:
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 00:00:00 EDT (1563249600); 29.279";  73.2F;  97%; 0.0 mph; N/A deg; 0.0 mph gust;  72.3F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:55:00 EDT (1563335700); 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-16 23:59:00 EDT (1563335940); 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 Using configuration file /usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:17:01 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Wed 17 Jul 01:17:01 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:17:01 EDT 2019 Date to check:                 2019-07-17
Wed 17 Jul 01:17:01 EDT 2019 Number of archive records:     17
Wed 17 Jul 01:17:01 EDT 2019 Number of WU records:          15
Wed 17 Jul 01:17:01 EDT 2019 Number of missing records:     3
Wed 17 Jul 01:17:01 EDT 2019
Wed 17 Jul 01:17:01 EDT 2019 Missing records:
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:15:00 EDT (1563340500); 29.173";  72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain  ...published.
Wed 17 Jul 01:17:01 EDT 2019 2019-07-17 01:16:00 EDT (1563340560); 29.173";  72.5F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  72.2F; 0.00" rain  ...published.
Wed 17 Jul 01:22:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped this iteration.
Wed 17 Jul 01:32:04 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped this iteration.
Wed 17 Jul 01:42:03 EDT 2019 WeeWX: Wunderfixer timestamp: (1563342124)
Wed 17 Jul 01:42:03 EDT 2019 Using configuration file /usr/share/weewx4/weewx.conf.
Wed 17 Jul 01:42:03 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Wed 17 Jul 01:42:03 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 01:42:03 EDT 2019 Date to check:                 2019-07-17
Wed 17 Jul 01:42:03 EDT 2019 Number of archive records:     22
Wed 17 Jul 01:42:03 EDT 2019 Number of WU records:          21
Wed 17 Jul 01:42:03 EDT 2019 Number of missing records:     1
Wed 17 Jul 01:42:03 EDT 2019
Wed 17 Jul 01:42:03 EDT 2019 Missing records:
Wed 17 Jul 01:42:03 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 01:52:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped this iteration.
Wed 17 Jul 02:02:04 EDT 2019 WeeWX: Wunderfixer timestamp: (1563343324)
Wed 17 Jul 02:02:04 EDT 2019 Using configuration file /usr/share/weewx4/weewx.conf.
Wed 17 Jul 02:02:04 EDT 2019 Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Wed 17 Jul 02:02:04 EDT 2019 Weather Underground Station:   KMIDEARB5
Wed 17 Jul 02:02:04 EDT 2019 Date to check:                 2019-07-17
Wed 17 Jul 02:02:04 EDT 2019 Number of archive records:     26
Wed 17 Jul 02:02:04 EDT 2019 Number of WU records:          25
Wed 17 Jul 02:02:04 EDT 2019 Number of missing records:     1
Wed 17 Jul 02:02:04 EDT 2019
Wed 17 Jul 02:02:04 EDT 2019 Missing records:
Wed 17 Jul 02:02:04 EDT 2019 2019-07-17 00:00:00 EDT (1563336000); 29.167";  72.0F;  99%; 0.0 mph; N/A deg; 0.0 mph gust;  71.7F; 0.00" rain  ...published.
Wed 17 Jul 02:12:03 EDT 2019 WeeWX: Wunderfixer has been intentionally skipped this iteration.


Regards,
\Leon
--
Leon Shaner :: Dearborn, Michigan (iPad)

On Jul 18, 2019, at 11:11 AM, Thomas Keffer <tke...@gmail.com> wrote:


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Thomas Keffer

unread,
Jul 18, 2019, 9:03:42 PM7/18/19
to weewx-user
I didn't touch how wunderfixer posts (uploads) the fixes, only how it downloads data.

-tk


Thanks, Rod!  =D

<div style=

Ray Mansell

unread,
Sep 7, 2019, 2:35:10 PM9/7/19
to weewx-user
Following an extended local internet outage yesterday, I found myself with several hours' worth of data missing from WU. I recalled wunderfix, tried it, encountered the 403 problem, and eventually discovered this thread. Having read most of it, I tried looking for the modified 3.9.1 version on Github, but it seems not to be there, unless I'm looking in the wrong place.

So, I wonder if someone would be so kind as to answer the following?

1. Is the 3.9.1 apikey-modified wunderfix still relevant? I.e. does it work?
2. If so, where might I find it?

Thank you.

Ray
Reply all
Reply to author
Forward
0 new messages