Schedule updates

[James Uppington]

Morning all,

Has anybody else noticed anything funny about the JSON schedule updates recently?

The URLs for the "Saturday Update" for 29/08 and the "Tuesday Update" for 08/09 both temporarily gave me a full schedule but were correct again after the event.

29/08's update returned a full schedule when my system downloaded it at 30/08/2015 01:05:57.

08/09's update returned a full schedule when my system downloaded it at 09/09/2015 00:48:31.

The first one I didn't notice until I had all my trains duplicated. Upon checking my logs, the update contained 479223 new schedules and no deleted ones.

Last night's one I noticed this morning it was taking a long time to process the schedule update and cancelled the processing before restarting it - the correct file was then downloaded at 07:19:40 from the same URL. Again, definitely a full schedule as the uncompressed file was about 2.7 GB.

I'll stick in some checks to make sure I don't end up processing this dud data but wondered if anyone else had seen this - I'd never seen it before the 29/08 blip.

James

[Peter Hicks]

Hi James

I've seen this happen before, but we never got to the bottom of it.

I have a support call arranged for Friday, I'll put this on the "issues to investigate" list.

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Uppington]

Thanks Peter.

Just need to work out the most efficient way of handing this now until the issue is fixed!

James

[Phil Wieland]

This happened to me on the morning of 30 August fetching "https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sat". and I had to back out all the duplicate trains.  I've never seen it before nor since.

The latest version of my software rejects an update if it doesn't contain any delete transactions, so hopefully I will be protected.

Cheers,

Phil

[Peter Hicks]

It’s a while since I’ve looked at the JSON schedules, but I think there’s a serial number field in there which increments daily.  If there is, you can check that the update you receive has a serial number of 1 greater than the last update you received.

[Phil Wieland]

The unwanted full file has the same serial number, so that doesn't help.  There is also a field called type but that always says "full" whether it's a full timetable or an update.  (Perhaps that means full as opposed to just one area?)

Phil

[Phil Wieland]

I should have added that when I repeated the download two hours later I got the correct file.

[Tom Lane]

> There is also a field called type but that always says "full" whether it's a full timetable or an update.  (Perhaps that means full as opposed to just one area?)

Looks like a new bug? Definitely recall being able to differentiate between the two by this field (type) in the past. This would have been on the Great Western (EF) updates feed. 

[Peter Hicks]

Nothing has changed on the schedule side for months... It may be a problem to do with timing. I'll check it out.

Meanwhile, a way to make sure you're not processing an old update is to record the file size and md5sum of the last week's worth of files. If you receive a file within a week that has the same size and hash as one you received earlier that week, you've received an old file.

It'd be useful if anyone getting old files could post about it here, along with the date and time you downloaded it. I'm thinking there may be a problem with timings somewhere.

Peter

[Phil Wieland]

Peter, I think you might be looking at the wrong problem?  I was attempting to get the updates file but received a full timetable file instead. 

Phil

[Peter Hicks]

Hi Phil

On 10 Sep 2015, at 09:34, Phil Wieland <goo...@philwieland.com> wrote:

> Peter, I think you might be looking at the wrong problem? I was attempting to get the updates file but received a full timetable file instead.

OK, that shouldn’t ever happen - they’re completely different files!

I’ll get it looked in to.


Peter

[Chris Bailiss]

Hi Peter.

May or may not be worth mentioning, but my CIF schedule download code appeared to have received a JSON file a few days ago. It was caught as the header checks obviously failed, recording an excerpt of JSON in the error message. Retried automatically an hour later and then got the CIF file.

Can't 100% rule out a bug on my side, but that version of the code has been running since April and has been stable.

Chris

[Dave Butland]

This sounds like the problem I see quite regularly with the CIF file. Sometimes the redirect points me to JSON files, sometimes it points me to the wrong CIF file. I've made my code very robust to check if the file has been seen before and if it is in the correct format. My suspicion is that it is something to do with the redirects (either caused by how my code handles it or by something on the aws side). Support were unable to reproduce the problem at their end despite me sending them code that definitely showed the problem at mine. 

[Peter Mount]

I had that problem yesterday.

The Cron job retrieved the cif as usual but got the json file instead.

The problem is it usually fixes itself within an hour or so, which might be why support couldn't reproduce it.

--

[Chris Bailiss]

Dave, Peter,

Thanks for the replies.  Matches my experience last week i.e. request CIF, get JSON, file fails client side validation checks on CIF header, etc, waits an hour or so, re-requests CIF file and this time gets the CIF file.

Glad it sounds like not a problem with my code.

Chris

[Peter Mount]

It does vary when it happens l. Last time before yesterday was the 3rd.

--

[Phil Wieland]

 I observed a different version of this this morning:

At 03:58 UTC I fetched "https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-sun", and I got the update file from two days ago, sequence number 1185.

My program rejected that and retried at 04:30 UTC when it received the correct update file, sequence number 1187 from the same URL.

I think I need to learn a bit more about how this is supposed to work ...

Phil

On Wednesday, September 9, 2015 at 11:27:03 AM UTC+1, James Uppington wrote:

[Chris Bailiss]

Had the problem of request CIF, get JSON again this morning.  Have raised a support ticket.

Chris

On Wednesday, September 9, 2015 at 11:27:03 AM UTC+1, James Uppington wrote:

[Peter Hicks]

On 15 Sep 2015, at 08:30, Chris Bailiss <cbai...@gmail.com> wrote:

> Had the problem of request CIF, get JSON again this morning. Have raised a support ticket.

Thanks - this is really useful. I’m meeting up with the support team today and tomorrow for another project, so I’ll bring this up.


Peter

[Phil Wieland]

At 03:58 UTC this morning, I fetched

https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-fri

And I received a full timetable file, not an update file.

A retry at 04:31 UTC the correct file arrived.

Do reports like this help the investigation?

Cheers,

Phil

On Wednesday, September 9, 2015 at 11:27:03 AM UTC+1, James Uppington wrote:

[Chris....@cancer.org.uk]

I can add a little to this...

Following a suggestion from Peter a couple of weeks ago, I added URL logging around the schedule downloads.

The problem occurred for me again this morning (for the first time since the logging was added). This showed that the redirect URL returned was not the extract asked for - it gave me a JSON extract from a different day:

Time: Tue 06 Oct 04:09:49.7
RequestURL = https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-mon.CIF.gz
Time: Tue 06 Oct 04:09:50.1
ResponseURL = https://nr-datafeed-cif.s3.amazonaws.com/CIF_ALL_UPDATE_DAILY/toc-update-sun.json?Expires=1444101079&AWSAccessKeyId=AKIAJGDB3M65TAQ2P2NQ&Signature=TsOQiH/ugXs1DoHKRicoEUB+6mw=

Peter, I opened a support call on this, but am travelling at the moment, so don't have access to that call number. Would you be able to attach the above information to it for me. Thanks.

Hope this helps

Regards

Chris

[Juhani Pirttilahti]

Hello,

My software requested monday's CIF update, but the CIF file it received is identical to thursday's update (2015-10-15). At least it wasn't in JSON format at this time.

[Peter Hicks]

Hi Juhani

This is an issue which has been reported many times before (genuinely, thanks for reporting it again!) - but one which I can't get support to fix because they can't reproduce it.  I'm not sure that's a valid argument though.

Could you drop the details of the problem to nrod.s...@rockshore.net, please?

Cheers,

Peter

On Tue, 20 Oct 2015 at 03:31 Juhani Pirttilahti <juhani.pi...@gmail.com> wrote:

Hello,

My software requested monday's CIF update, but the CIF file it received is identical to thursday's update (2015-10-15). At least it wasn't in JSON format at this time.