Changes to CIF data

[Peter Hicks]

All,

There are some changes to the CIF data format that are being proposed by Network Rail.

Many of them are inconsequential and are just changes to documentation, but as an early heads-up, the following may affect you if you consume CIF data from the Network Rail Data Feeds platform:

* The train identity field is changing to be able to support four-digit train identities as well as the existing number-letter-number-number format
* There will be a new activity code, ‘AX’ - “Passes another train on arrival”

I don’t have a date for these changes taking effect yet, but when I do, I’ll communicate it accordingly.


Peter

[Chris Bailiss]

Hi Peter

Thanks for the heads-up.

Is there any downloadable documentation available from Network Rail about this?  e.g. a lot of the CIF documentation can be downloaded in PDF form, so just wondering if there are any PDFs or similar about these proposed changes.

Also, while we are on the topic of CIF, have there been any developments in the discussions about the scheduling / creation times for the CIF files? - to eliminate the delayed updates that seem to happen sometimes. 

Thanks

Chris

[Peter Hicks]

Hi Chris

On 27 Jan 2015, at 10:09, Chris Bailiss <cbai...@gmail.com> wrote:

> Is there any downloadable documentation available from Network Rail about this? e.g. a lot of the CIF documentation can be downloaded in PDF form, so just wondering if there are any PDFs or similar about these proposed changes.

There’s an updated spec and a list of changes, but at the moment it’s at the consultation stage so I can’t distribute it. However, once the CIF Specification is formally reissued, I’ll update the copy on the wiki.

> Also, while we are on the topic of CIF, have there been any developments in the discussions about the scheduling / creation times for the CIF files? - to eliminate the delayed updates that seem to happen sometimes.

There have been - there are two options we’re looking at:

1. Produce JSON data at 2200 (or 0001) each day regardless of whether a CIF file has already arrived - so there should always be JSON data available by a certain time every night, even if it’s empty
2. Produce JSON data as soon as the CIF file has arrived, which will result in JSON data being available at some time or, if there was a problem generating the CIF file on Network Rail’s side, no data

Both of these have their plus and minus points, and both are under consideration. After doing some analysis, there are only a very small number of ‘A for B’ schedules that come through on CIF, so producing JSON at the same time each day may, if the file arrives after that time, result in one or two trains for which there’s no schedule.

What’s your preference? And if anyone else is reading who consumes CIF, what preference do you have?


Peter

[petermount]

As someone who consumes the CIF (rather than the JSON one) there's two common issues that cause the update job to fail:

1. The CIF hasn't been received on time so when I download the file it's still showing the previous week's daily update not the current one
2. sometimes the download fails as I then receive the JSON file rather than the CIF. Doesn't happen often but does happen a couple of times a month (roughly)

So for me as long as it's available when it arrives would be fine - the job is currently running at 0600 to allow for late arrivals, but even then the CIF file sometimes isn't there which may cause problems for a missing schedule first thing in the morning.

Peter

[Dave Butland]

Peter, 

I'm not entirely clear what you mean by producing the JSON file before the CIF. I'd always assumed the JSON file was just a transformation of the CIF file. Is this not the case? I'd rather not get an empty JSON file though if the CIF hasn't arrived. How would we get updates or know that updates were now available?

My preference is to produce the JSON file as soon as the CIF file is available. I check the file at 6am at the moment, read the header and if it isn't the new file then I reschedule every ten minutes for an hour until it is. That allows the file to be an hour late before I raise an alert.

My real preference is to have it published on a topic though as then I don't have to poll for it and it arrives as if by magic. Appreciate this might be more work for you though and my polling code is already well tested. 

Dave.

[Steve Ardagh-Walter]

Hi Peter

My preference would be to only post JSON when the CIF has arrived - that would make it simpler to raise an alert when a (meaningful) file is not available.   I don't see the value of an empty JSON file.

Another factor is how early JSON can be published when everything is working 'normally'.   I think the wiki currently states that JSON is available by 0600 which presumably builds in a fair bit of padding.      This is too late for any schedule changes for that day, so I'm currently processing the previous day's file at 0330 and then adding any changes after the fresh file at 0600.     Midnight or 2200 would be great - how often should one of those times be achieved ?

Steve 

On Tuesday, 27 January 2015 11:21:32 UTC, Peter Hicks wrote: