Download Issues

[Paul Lockwood]

Are there issues with getting the XML files?

It started this afternoon and still not working properly where I only get the site header details as below.

<SiteRep>

<Wx>

<Param name="F" units="C">Feels Like Temperature</Param>

<Param name="G" units="mph">Wind Gust</Param>

<Param name="H" units="%">Screen Relative Humidity</Param>

<Param name="T" units="C">Temperature</Param>

<Param name="V" units="">Visibility</Param>

<Param name="D" units="compass">Wind Direction</Param>

<Param name="S" units="mph">Wind Speed</Param>

<Param name="U" units="">Max UV Index</Param>

<Param name="W" units="">Weather Type</Param>

<Param name="Pp" units="%">Precipitation Probability</Param>

</Wx>

<DV dataDate="2014-02-26T21:00:00Z" type="Forecast"/>

</SiteRep>

[Henry Hunt]

Are you using a site ID in the URL request from the correct site list? Only sites from the hourly observations site list will return valid hourly observation data so sites from the daily and three hourly forecast won't return anything except the header like above.

[Paul Lockwood]

Yes I am using the normal full URL that is required.

I have created (creating) an app that requires to download around 105 different sites all at the same time when it is required to get the updated forecast. It was working fine until late yesterday where the site ID have not been changed in my list for a few days. I had the same issue last night from home.

I've checked this morning and it looks like it is working again. If there was an issue it would be nice to have a status page to advise.

http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/3210?res=3hourly&key=(MYAPI removed)

On Thursday, 27 February 2014 08:08:44 UTC, Henry Hunt wrote:

Are you using a site ID in the URL request from the correct site list? Only sites from the hourly observations site list will return valid hourly observation data so sites from the daily and three hourly forecast won't return anything except the header like above.

On Wednesday, February 26, 2014 11:20:15 PM UTC, Paul Lockwood wrote:

[Paul Lockwood]

It has done it again. Downloaded all required this morning, if I run it now I get about 8 or 9 proper downloads and about 80 failed as per the example at the start of the post.

[sean arrowsmith]

Same with me. Been fine for 8 months, then a few days ago I am just getting an empty xml file 99.9% of the time and not stable at all.


Agree with getting a status page.

Was using a chinese site for data called Timer7 but they have become very unreliable over the last few months.

[Paul Lockwood]

It seems to be afternoons and evenings in the UK that the data becomes unstable. I've had to write in my code some checking on the file size so at least I have some old data if the new doesn't download.

I've downloaded a couple of apps on my Windows and android phone that should use the 7Timer data but so far over the last two weeks haven't had any data come through, very disappointed.

[Paul Backhouse]

I'm seeing exactly the same thing right now for the following url:

http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/351671?res=3hourly&key={key}

No ['Locations']['Location'] in either json or xml report.

I saw it earlier (about an hour ago) for the daily forecast but it seemed to get fixed.

[MetOffice DataPoint]

I am raising an incident to get this looked into, please keep an eye out for a notification with an update to what the problem is.

 

Thanks

 

Clare

 

 

PS, we are working on implementing a status page.

[sean arrowsmith]

Thank you Clare

PS It is still happening, I use an hourly schedule to pick up data and out of yesterdays 24 slots I only managed 3 downloads, these at 03:05, 10:05 and 13:05

[sean arrowsmith]

Yesterday everything worked fine from 1am to 2pm, then nothing worked all afternoon until 1am this morning

The url for the xml I collect is
datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/xml/351670?res=3hourly&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

When it fails this is the xml I receive back from the php call

<?xml version="1.0" encoding="UTF-8"?>

<SiteRep>
<Wx>
<Param name="F" units="C">Feels Like Temperature</Param>
<Param name="G" units="mph">Wind Gust</Param>
<Param name="H" units="%">Screen Relative Humidity</Param>
<Param name="T" units="C">Temperature</Param>
<Param name="V" units="">Visibility</Param>
<Param name="D" units="compass">Wind Direction</Param>
<Param name="S" units="mph">Wind Speed</Param>
<Param name="U" units="">Max UV Index</Param>
<Param name="W" units="">Weather Type</Param>
<Param name="Pp" units="%">Precipitation Probability</Param>
</Wx>

<DV dataDate="2014-03-02T23:00:00Z" type="Forecast">
</DV></SiteRep>

[sean arrowsmith]

Failed 6 times yesterday and surprising only 3 times today.

Could it be a problem with PHP which I use to pull the XML off datapoint?

[colin.corkerton]

Hi Sean,

I've checked with our technical team and they advise the use of PHP is not the problem. We have found the cause of the issue and a fix is currently being tested. It will be released as soon as possible.

Colin

[Paul Lockwood]

Certainly not PHP. I use an AutoIT script to save the URL data page which uses IE.

From there I then use the same script to read in all the data into an array and use as required.

Any details I need to keep as a comparison etc. I then write to another XML file.

My script looks for up to around 105 sites to download at once, so that I can produce my own layout of local forecast sites in a circular basis from a selected location in 20 KM steps up to 80 KM away.

Being into Astronomy and our wonderful UK weather I need to see what all the local forecasts are so that I can see if just down the road the weather is different and see where the weather is moving.

[sean arrowsmith]

On Monday, 17 March 2014 10:39:20 UTC, Paul Lockwood wrote:

Being into Astronomy and our wonderful UK weather I need to see what all the local forecasts are so that I can see if just down the road the weather is different and see where the weather is moving.

Paul, that is what I use it for. It would be nice to see some Met Office atmosphere seeing data. 7Timer can produce this data but I have no idea how it is done or the formula if it is calculated from from several other datapoints.

[Mark G]

Downloads have been intermittent for a while, but as of 11am this morning all of the feeds are returning a 403 Forbbiden.

I have checked that this is not unique to this API key (which it is not), and have made sure the HTTP Request Header is correct.

Is anyone else getting this issue?

On Wednesday, February 26, 2014 11:20:15 PM UTC, Paul Lockwood wrote:

[Tim Welford]

Yes, I am getting this 403 Forbidden issue, I have tried several work arounds

I have emailed enquiries ***@***  metoffice.gov.uk who have taken support requests to the service desk for me before.

I will post back here if \ when I get a response, it normally takes a while.

[Brian Gaze]

Yes there seems to be a problem at the moment. The last radar update I had was 11:30GMT. 

--
You received this message because you are subscribed to the Google Groups "Met Office DataPoint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metoffice-datap...@googlegroups.com.
To post to this group, send email to metoffice...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Welford]

I had an email back:

Thank you for your email.
This issue should be fixed now, please let me know if it is still occurring. 

 And it does seem to be coming back onlin.

A status page should be a doddle, I can't see why it has taken months.

I may knock one up myself for us all to use.

[Mark G]

Thanks Tim -- I can confirm that it's now working this end. Resolved.

[sean arrowsmith]

Sorry Mark but the issue we are having with these download is not what this hijacked thread was about. The issue this thread is talking about is that during the afternoon when we request the xml for forcast we are given an empty xml.

This is still going on Met Office, again every php request I make pulls an empty PHP all afternoon and when it come to midnight all seems well until we hit midday again

On Thursday, 20 March 2014 13:05:55 UTC, Mark G wrote:

Thanks Tim -- I can confirm that it's now working this end. Resolved.

On Thursday, March 20, 2014 1:03:53 PM UTC, Tim Welford wrote:

[Mark G]

Hi Sean,

Sorry if we hijacked your thread!

This is a long running issue for me also, I have just built in a try-based class which retries until it's pulled it down - if it fails 10 times in a row then an email is sent to me to prompt a manual pull - however I am still getting failures (and thus email prompts).  To confirm I also use PHP as the interface medium, and pull XML.

If you can get to the bottom of this issue then it would also help me greatly.

Regards,

M

[sean arrowsmith]

It's alright.

I am still waiting for the fix colin.corkerton posted about 10days ago to be implemented. It can not take that long to fix a simple xml file generation

On Tuesday, 25 March 2014 08:47:00 UTC, Mark G wrote:

Hi Sean,

Sorry if we hijacked your thread!

This is a long running issue for me also, I have just built in a try-based class which retries until it's pulled it down - if it fails 10 times in a row then an email is sent to me to prompt a manual pull - however I am still getting failures (and thus email prompts).  To confirm I also use PHP as the interface medium, and pull XML.

If you can get to the bottom of this issue then it would also help me greatly.

Regards,

M

On Tuesday, March 25, 2014 8:08:28 AM UTC, sean arrowsmith wrote:

[colin.corkerton]

Hi all,

The release today included a fix for this issue. Therefore this problem should not now occur. However please let us know if you do still encounter this issue by contacting  enqu...@metoffice.gov.uk  including DataPoint in the subject line.

Regards,

Colin 

[sean arrowsmith]

last failure was 27/03/2014 @ 7:05am for me.

It has been working great since them. Thank you Colin and your Team. My forecasting for astronomy usage can now continue. Thanks
Sean

[hazel-bill]

I know that a lot of water has passed under the bridge since your post, but I was looking at my post to see if I had had any response and I spotted your Post.  I had the very same problem and e-mailed the weather desk about this on the 9^th Jan and after lots more e-mails backwards and forwards I got this reply on the 29^th Jan:

Site ID 99062 is not an observation site. 

For all your wxobs requests, please refer to the list of observations sites. http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/xml/sitelist?key=. 

While there are over 5000 forecast sites, there are only 132 observations sites

I tried the above link expecting there to only be 132 sites, but there were a lot more and some of them did not contain observation data. I also had had problems trying to access attributes from Child Nodes greater than the 128^th Child Node.  I e-mailed the Weather desk about this and there eventual reply on the 21^st Feb was to Post my question where they would ask there Web Team to investigate the matter further.  You can see my post to which I have no response to since.

 

On Wednesday, 26 February 2014 23:20:15 UTC, Paul Lockwood wrote: