Cant download Uk Met Office forecast

[Phil Owers]

Hi Guys
Have 2 raspberry pi each running to a vantage pro. Both stopped downloading uk met office forecasts (2 api keys) on the 25 feb. The log suggests HTTP Error 403 Forbidden. If i put the http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/354731?res=3hourly&key=api key into a browser it has data. Has anybody else had this problem. Nothing has changed on the rasberry as both have been working without intervention since 3.9.2 many weeks ago. Thanks for any help Phil

[vigilance wx]

Hi

Yes my link to UK met office is also down 

cant get to the Pi  at the moment to find out why

[Ben B]

Hi all

Yes I have this 403 issue too - checked my Met Office account and no issue there as account and API key active and nothing changed on the Pi or weewx. Using weewx v3.8 and Forecast 3.3.2.  Need some help please!

forecast: UKMOThread: UKMO: download forecast from 'http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/354123?res=3hourly&key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX8995'

forecast: UKMOThread: UKMO: failed attempt 1 to download forecast: HTTP Error 403: Forbidden

Thanks, Ben.

[Phil Owers]

I was in contact with enqu...@metoffice.gov.uk yesterday but they thought it was more my end than theres.  Now others are reporting the same issue perhaps if we all send them an email to the above address they might start to think its there end. I did create a new account with a new API but also failed.  Unless of course somebody has another theory.  Phil

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6cd01ffe-a138-4ea9-b95b-599aa4fe3441%40googlegroups.com.

[Phil Owers]

This is the first few lines of what I get back from the datapoint url on my browser

{"SiteRep":{"Wx":{"Param":[{"name":"F","units":"C","$":"Feels Like Temperature"},{"name":"G","units":"mph","$":"Wind Gust"},{"name":"H","units":"%","$":"Screen Relative Humidity"},{"name":"T","units":"C","$":"Temperature"},{"name":"V","units":"","$":"Visibility"},{"name":"D","units":"compass","$":"Wind Direction"},{"name":"S","units":"mph","$":"Wind Speed"},{"name":"U","units":"","$":"Max UV Index"},{"name":"W","units":"","$":"Weather Type"},{"name":"Pp","units":"%","$":"Precipitation Probability"}]},"DV":{"dataDate":"2020-03-01T10:00:00Z","type":"Forecast","Location":{"i":"354731","lat":"52.9456","lon":"1.212","name":"SHERINGHAM (BEACH)","country":"ENGLAND","continent":"EUROPE","elevation":"0.0","Period":[{"type":"Day","value":"2020-03-01Z","Rep":[{"D":"SW","F":"-2","G":"40","H":"78","Pp":"0","S":"25","T":"4","V":"GO","W":"0","U":"0","$":"360"},{"D":"SW","F":"-1","G":"40","H":"72","Pp":"0","S":"27","T":"5","V":"VG","W":"1","U":"1","$":"540"},{"D":"SW","F":"2","G":"38","H":"63","Pp":"1","S":"25","T":"7","V":"VG","W":"1","U":"2","$":"720"},{"

Can anybody spot any changes that may give us the HTTP 403 Error.

Thanks Phil

On Saturday, February 29, 2020 at 6:38:50 PM UTC, Phil Owers wrote:

[Phil Owers]

The UK Met Office has now raised an issue with there datapoint services but did suggest to try

Could you try the following : change the User-Agent to
'Mozilla/5.0'. It could be that the user agent of 'python/urllib/2.5' is the issue. This worked for some users.

Not sure what that means

Phil

On Saturday, February 29, 2020 at 6:38:50 PM UTC, Phil Owers wrote:

[Mike Hawkins]

I can confirm that this does work, I modified the download function in forecast.py to:

                req = urllib2.Request(u)
                req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36')
                response = urllib2.urlopen(req)
                return response.read()

Mike

[mwall]

On Sunday, March 1, 2020 at 7:17:00 AM UTC-5, Phil Owers wrote:

The UK Met Office has now raised an issue with there datapoint services but did suggest to try

Could you try the following : change the User-Agent to
'Mozilla/5.0'. It could be that the user agent of 'python/urllib/2.5' is the issue. This worked for some users.

if they are limiting access based on user-agent, then uk met office is using some unconventional (i might venture to say downright stupid) programming practices.  adjusting the response of the web server based on the user agent makes sense in some user-facing corner cases, for example, when microsoft won't fix internet explorer 6.  but there are much better approaches.  and doing that it for a machine-to-machine interface is downright silly.

if you hack the forecast code, use the bare minimum necessary to make it work.  i would suggest simply 'Mozilla/5.0', not a full user-agent string with platform information.

hopefully uk met office will fix their systems...

m

[Phil Owers]

Hi Mike

Sorry to be a pain but did you modify or add and if you added could you explain where.

Phil

On Saturday, February 29, 2020 at 6:38:50 PM UTC, Phil Owers wrote:

[Mike Hawkins]

Hi, yes, sorry the last email was a bit of a dump and run as I was in a hurry.

I have attached a forecast.py file with the two added and one modified line.  This is at line 3280.  I have changed

            try:
                response = urllib2.urlopen(u)
                return response.read()

to:

            try:
                req = urllib2.Request(u)
                req.add_header('User-Agent', 'Mozilla/5.0')

                response = urllib2.urlopen(req)
                return response.read()

the file to edit or replace is user/forecast.py.  The shorter user-agent seems to work just as well as the longer one.  

Hope it works for you

Mike

[Phil Owers]

Thanks Mike that worked.

Strange way to go about things though from the Met Office. They did tell me Datapoint is to be replaced by DataHub. Perhaps its there way of changing.  Thanks again Phil 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6ef7dbb2-b6cf-45b6-9eb8-dc07cef5930c%40googlegroups.com.