403 Forbidden

[Paul Backhouse]

Hi,

I'm getting 403 Forbidden when requesting data (observation / forecast) from DataPoint. This has been occurring for a couple of days.

Any one know why?

Thanks,

Paul

[Steve Houghton]

Hi Paul

Did this resolve itself for you?  I started getting them yesterday but only when called from my AWS server.  It works fine in a browser from home or work with the same API key.

I wondered if they had started blocking certain IP addresses for some reason.

thanks

Steve

[b4r...@gmail.com]

Hi Steve,

I'm having the exact same problem. 403 Forbidden when trying to access the 3-hourly forecast data.

It works fine from my local web browser though.

UPDATE: I can also download the maps (forecast and radar precipitation) from my server, just not the 3-hourly forecast data by the looks of it.

UPDATE 2: I also cannot download the data from other linux servers via wget, I've tried 3. Works from browsers, but not linux server. Looks like it's a data supply issue, rather than an IP address issue.

Did you find a solution? Have you reported the problem?

Andrew

[Steve Houghton]

Hi Andrew

I tried on a second AWS instance with the same result and both from python and curl  (not sure if they call the same low level functions or not.....)

I didn't report initially I was worried I may have gone over quota but it doesn't seem this is the cause so I have reported it tonight.

regards

Steve

[b4r...@gmail.com]

Hi Steve,

I've actually found a workaround now. I was also using python, but the urllib2 package. For me, calling wget from within python now works.
Here is my working code. Hope it helps.
Out of interest, what are you using the data for - is it part of a public website/app?


import urllib2 as url_lib
import json

url='http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/all?res=3hourly&key='+api_key
try:
    page = url_lib.urlopen(url) #This was my original method
except:
    print "Normal url_lib method failed, try wget instead..."
    import subprocess as sub
    location = download_location
    args = ['/usr/bin/wget', '-P', location, url]
    output = sub.Popen(args)
    output.wait()
    newurl=location+'/all?res=3hourly&key='+api_key
    page = url_lib.urlopen('file://'+newurl)

pg=page.read()
x = json.loads(pg)

[Steve Houghton]

Thanks Andrew

I will try when I get chance (and if my support request doesn't sort it!)

I am just using it for hobby use - it started as a way to validate my weather station and then I quickly realised there wasn't much point in running my own weather station so I am now just using it as a way to learn about various technologies.  I am storing the data in an AWS database and am now learning how to use DJango to interact with it.  I've got an idea for a project to use machine learning on the data to work out when I can and can't trust their forecast.

How about you?

regards

Steve

[Jonathan Stonehouse]

I'm experiencing the same problem when accessing site data, forecast capabilities, and 3-hour forecasts. Like you, it works from my browser but my Python script returns 403 Forbidden. I'm using urllib so my code is similar to your own. My logs indicate this started on the 6th so this is the third day without data.

JS

On Thursday, 7 September 2017 15:15:51 UTC+1, b4r...@gmail.com wrote:

[Jonathan Stonehouse]

I tried your wget workaround and it works. Be interesting to see what the Met's support team have to say about the 403. I've been using the same Python script for approx. 9 months without any problems (other than occasional niggles due to server downtime).

JS.

On Friday, 8 September 2017 08:38:48 UTC+1, b4r...@gmail.com wrote:

Hi Steve,

I've actually found a workaround now. I was also using python, but the urllib2 package. For me, calling wget from within python now works.
Here is my working code. Hope it helps.

[Steve Houghton]

It took a while to convince them it wasn't just user error but it has been escalated so hopefully it will get sorted.

I will post feedback here

--
You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.
To post to this group, send email to metoffice...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Stonehouse]

My python script is working again so looks like they've sorted it.

To unsubscribe from this group and all its topics, send an email to metoffice-datapoint+unsub...@googlegroups.com.

[Steve Houghton]

Confirmation below from their support desk that they made a breaking change.

Steve

Dear Steve,

Following on from your initial email,  I can confirm that there was a change which broke this on Tuesday. It has now been reverted and we'll take Datapoint into consideration when we roll it forward again.

I’m sorry for any inconvenience caused.

 

To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.

To post to this group, send email to metoffice...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.

[Will Cooke]

On Sunday, 10 September 2017 09:16:17 UTC+1, Steve Houghton wrote:

Confirmation below from their support desk that they made a breaking change.

Steve

Dear Steve,

Following on from your initial email,  I can confirm that there was a change which broke this on Tuesday. It has now been reverted and we'll take Datapoint into consideration when we roll it forward again.

I’m sorry for any inconvenience caused.

I had the same problem, and it fixed itself (courtesy of you guys logging an issue), and now it's suddenly back again.

Can anyone confirm they've got the 403 error back again?

Cheers, Will

 

[Steve Houghton]

Hi Will

Yes just noticed it had stopped, but it seems okay again now?

Steve 

--

[Will Cooke]

Ah, yes - working again.

I'll be more patient next time.

Cheers!

On 15 September 2017 at 13:14, Steve Houghton <steve.h...@gmail.com> wrote:

Hi Will

Yes just noticed it had stopped, but it seems okay again now?

Steve 

On Fri, 15 Sep 2017, 11:42 Will Cooke <wi...@whizzy.org> wrote:

On Sunday, 10 September 2017 09:16:17 UTC+1, Steve Houghton wrote:

Confirmation below from their support desk that they made a breaking change.

Steve

Dear Steve,

Following on from your initial email,  I can confirm that there was a change which broke this on Tuesday. It has now been reverted and we'll take Datapoint into consideration when we roll it forward again.

I’m sorry for any inconvenience caused.

I had the same problem, and it fixed itself (courtesy of you guys logging an issue), and now it's suddenly back again.

Can anyone confirm they've got the 403 error back again?

Cheers, Will

 

--
You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to metoffice-datapoint+unsub...@googlegroups.com.
To post to this group, send email to metoffice-datapoint@googlegroups.com.

[Andy Turner]

Hi,

As of yesterday, I've just started having this problem. I've read the thread, but I'm not sure what to do, so I thought I would just mention the issue and have a think...

Andy

 

On Monday, 31 July 2017 21:25:52 UTC+1, Paul Backhouse wrote:

[Andy Barnard]

1 year on (minus 1 day), same here.  As of yesterday my linux scripts stopped working and:

- wget  http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/3840?res=3hourly&key=<my_key> returns 403 forbidden

- curl -o output_file http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/3840?res=3hourly&key=<my_key> returns no data

- http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/3840?res=3hourly&key=<my_key> in a browser from a windows PC produces an xml response

Not sure what to do either.  Just wait for it to start working as silently as it stopped? 

Andy

[Orange Bucket]

Several people seem to be experiencing 403 errors, as evidenced by the posts in three threads now, with some like myself having been seeing issues for about a week. There seem to be two camps, unless I am misinterpreting what people have been saying.

Some seem to be finding server-side scripts or command line tools (e.g. wget or curl) consistently fail with '403 Forbidden', but they find the URL works fine when accessed from a web browser.

Others, like me, seem to be seeing 'random' failures of their server side scripts. In this case direct testing with a web browser would not be particularly diagnostic as running the script again will almost invariably be successful.

I see the issue mostly with the 3 hourly 5 day forecasts but it has happened with the regional text forecasts too. I fetch the feeds one after the other in the same script and have yet to see both fail at once.

[Andy Turner]

Just so you know, I've hit the same issue again recently. The last time my scripts successfully worked was around about the 11th of January...

--
You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.
To post to this group, send an email to metoffice...@googlegroups.com.

[Andrew Fernando]

We are also getting a 403 intermittently and although if everyone's "fix" is to retry until a valid response is received, should the system already be under load, it is only going to exacerbate the problem further. I feel it would be better all round if Metoffice could check their infrastructure before everyone resorts to retrying as a solution.

[Dylan]

I've now noticed that it does indeed seem to be random. It failed twice today on my webserver, once on my local machine and then worked fine the second time.

On Wednesday, 23 January 2019 12:39:44 UTC, Orange Bucket wrote:

[Andrew Barnard]

For me its not random.  Wget and curl fail 100%, web works mostly 100%.

 

Something changed in last 24hrs though.  When I tried to ping datapoint.metoffice.gov.uk day before yesterday I never got the same IP two requests on the trot, there were more than a dozen unique responses.  Just now I get just two, 23.62.2.80 and 82.  All IPs are from Akamai.

 

Not sure if that is relevant, just an observation.

 

Sent from my mobile phone

--
You received this message because you are subscribed to the Google Groups "Met Office DataPoint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metoffice-datap...@googlegroups.com.
To post to this group, send email to metoffice...@googlegroups.com.

[Orange Bucket]

It could well be relevant if certain servers aren't serving the feeds correctly and DNS load balancing is being used. Different nameservers give different IP addresses and if their cache is cleared they are different again.

[Andy Turner]

Hi,

I'm getting a HTTP 403 error again. It worked fine this morning. I'll try again tomorrow to see if the issue is somehow magically resolved overnight... 

--
You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.

[Campbell Morrison]

Yes, I'm getting them as well - still happening today.

On Monday, 15 April 2019 19:49:50 UTC+1, Andy Turner wrote:

Hi,

[Orange Bucket]

Apologies for a 'me too'. I'm only checking once per hour but that is sufficient to see that there are intermittent 403 errors on both current obs and three hourly forecasts.

[Andy Turner]

Still an issue for me. I'm doing a twice daily check. Hopefully there will be some information about the problem soon or it will just start working again...

On Wed, Apr 17, 2019 at 8:09 AM Orange Bucket <anid...@gmail.com> wrote:

Apologies for a 'me too'. I'm only checking once per hour but that is sufficient to see that there are intermittent 403 errors on both current obs and three hourly forecasts.

--
You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.

To post to this group, send an email to metoffice...@googlegroups.com.

[Martin Watson]

Me too. Random failures from php script on server, fine from local browser for 5 day forecast.

I've only been using datapoint for a week since weather underground withdrew their feed, so I am glad to have found this thread and that it isn't my code that's at fault! Still, an inauspicious introduction to datapoint for me.

Martin 

[Andy Turner]

Hi Martin, MODP list,

I think DataPoint has been fairly reliable, but this is now a long period of failure. I had hoped it would be working again by now, but sadly not.

Is there anything else we can do to ask about timescales for getting this working again?

Best wishes,

Andy

--

You received this message because you are subscribed to a topic in the Google Groups "Met Office DataPoint" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metoffice-datapoint/oCFiAHxJNlY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metoffice-datap...@googlegroups.com.

To post to this group, send email to metoffice...@googlegroups.com.

[Campbell Morrison]

I raised this as an incident (INC000045201) with the Met Office service desk on 17 Apr, but haven't had a response yet (apart from requests for more information).

[Campbell Morrison]

I've just had the following response from the Met Office: "Our support team have restarted one of the service and have asked to get confirmation that this has been resolved?"

Is anyone still getting errors?

[Andy Turner]

Thanks Campbell. How did you raise the incident with the Met Office service desk?

I was getting errors at 9am and am still getting the same error when I tried just now. Many thanks for your help. 

On Wed, Apr 24, 2019 at 6:26 PM Campbell Morrison <campbell...@gmail.com> wrote:

I've just had the following response from the Met Office: "Our support team have restarted one of the service and have asked to get confirmation that this has been resolved?"

Is anyone still getting errors?

[Campbell Morrison]

Yes, using https://www.metoffice.gov.uk/forms/report-technical-fault

Make sure you (a) quote the email address to which your API key is registered and (b) include a screenshot of the 403 error - assuming you can reproduce one in a browser.   Just saying that you get a 403 error isn't sufficient to keep the system happy!

[Andrew Fernando]

Hi all,

Just to mention we don't seem to be getting any of these errors you are experiencing so am confused why only some people are getting them.

Could you add a bit of detail in your methods? Language, methods used etc. Perhaps someone can help figure out the issue you are experiencing.

Regards,

Andy

You received this message because you are subscribed to the Google Groups "Met Office DataPoint" group.

To unsubscribe from this group and stop receiving emails from it, send an email to metoffice-datap...@googlegroups.com.

[Campbell Morrison]

I'm getting the 3-hour forecast using PHP:

    $context = stream_context_create(array(
        'http' => array(
            'protocol_version' => 1.1,
            'user_agent' => 'PHP',
            'timeout' => 3,
            'method'  => 'GET',
            'header'  => 'Connection: close')
          )
      );
     
    $contents = file_get_contents($url, false, $context);

However, I can also get a 403 when I use a browser.  The problem is that the 403 errors don't occur every time, so it's quite difficult to reproduce one in a browser, but I see lots in my PHP error log.   The Met Office seem to have fixed the problem now for me, though others are still suffering.

[Martin Watson]

I've been error-free for a few days now. Looks like it's fixed from here.

Martin

[Andy Turner]

Hi Andy, all,

I use Java as per the example code below (substitute your API key as KEY below to get this to work - or maybe not):

String url = "http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/capabilities?res=3hourly&key=" + KEY

URL u = new URL(url);

HttpURLConnection c = (HttpURLConnection) u.openConnection();

I get the response code using the following (and then check it):

int responseCode = c.getResponseCode();

[Andy Turner]

It is working again for me now and probably would have worked a while ago if not for a bug I introduced into my code coincidentally at the same time I started getting the 403 error and not being able to test the changes I made to things. Basically the bug resulted in me reading only a partial part of my API key from a text file I had used to store it. So I am sorry as my code then might have been sending lots of requests to the service with a wrong API key which may have compounded the problem. It would have done this quite a lot! I count and limit the number of requests, but still, this could have made things worse for everyone when my program was running.

Many thanks for your help.

Best wishes,

Andy

[Andrew Fernando]

Glad you managed to figure it out.

All the best,

Andy

[Campbell Morrison]

The 403 errors seem to be back.  I've raised an incident report with the Met Office.

[Kev Waite]

Hi. I found this thread while digging into why my Raspberry Pi app that uses the API had recently failed.  The 403 Forbidden response was the culprit.  The URL worked fine directly from a browser.  After some experimentation I pinned it down to the User Agent header being sent.  The default one in Python's urllib module is 'Python n/m' (n/m being the version).  I changed the header to be something like what a browser sends and the call started working.  I hope this helps.

Cheers,

   Kev

[Ben Walters]

Funnily enough I was about to post the same thing. I got the solution from this SO post.

In Lib/site-packages/metoffer.py I replaced line 186 from:

page = url_lib.urlopen(url)

to:

req = url_lib.Request(url, headers={'User-Agent': "Magic Browser"})

page = url_lib.urlopen(req)

and it's working perfectly.

[Campbell Morrison]

I think you could be right.  I haven't had any 403 errors since Sept 2019 when I changed my user agent string in the curl request from 'PHP' to the user's user agent: $_SERVER['HTTP_USER_AGENT'].  I hope I haven't spoken too soon!

[Don P]

I also have had occasional 403 failures with the 3 hour forecast over the years, but they usually go away after 24 hours.  This time, they have not.  As you guys report, the call works in a browser.  I changed my Raspberry Pi Python 3 call from:

      response = urllib.request.urlopen('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/354037?res=3hourly&key=................

to:

      response = requests.get('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/354037?res=3hourly&key=...............

and this works fine, even though the urllib call does not.

I also deleted (obviously):

      import urllib.request

and replaced it with:

      import requests

I am an amateur Python programmer and do not know the reasons for the above.  However, I hope this helps.

[Campbell Morrison]

I think you could be right.  I haven't had any 403 errors since Sept 2019 when I changed my user agent string in the curl request from 'PHP' to the user's user agent: $_SERVER['HTTP_USER_AGENT'].  I hope I haven't spoken too soon!

Yes, I spoke too soon!  I'm getting 403 errors again, despite having a browser like user agent string.  I even get the 403 error if I enter the URL in a browser.  I'll raise an incident with the Met Office. 

[Stephen Murray]

Thanks for raising the incident. I was just checking here to make sure if anyone else had done so already!

A user pointed out the issue to me. I have always used the default user agent header for Python, but have resorted to using something else. I am not happy with having to do so. Can only guess they've been tinkering with their firewall.

[Campbell Morrison]

Well, I raised the incident but they stopped happening and I couldn't reproduce the problem so the incident was closed.

However, I subsequently found that although I thought my code was using a browser-like user agent string, in fact it wasn't.  I've changed it to do so and we'll see what happens.

[Campbell Morrison]

Well, I raised the incident but they stopped happening and I couldn't reproduce the problem so the incident was closed.

However, I subsequently found that although I thought my code was using a browser-like user agent string, in fact it wasn't.  I've changed it to do so and we'll see what happens.

I'm still not convinced though that that's the solution.  When the 403 errors do occur I am able occasionally to reproduce the problem by entering the URL directly into the address bar of a browser.