Latest RDG Timetable Feeds

[Michael Flynn]

The Timetable Feeds on the Data Download page at RDG's http://data.atoc.org/ site aren't updating as per normal. I normally download the dataset on a Saturday for the latest schedule but last week the set (ttis951.zip), according to the header records, were updated to the 21st of March whereas I would have expected it to be to the Friday, i.e., the 26th and this week there is no new file available, the Current Timetable Feed is still last week's (27/03/2021, ttis951.zip). There doesn't appear to be any information as to why this might be.  Have I missed an announcement?  Nor have I noticed anything on the forum here. Anyone on the forum able to shed light?  I haven't yet contacted the support team but will do if need be.

I am aware that there has been or are other ways to access the data but I haven't kept a note of the details. I'd be grateful if someone was able to point me in the right direction, though I'm guessing if the data hasn't been updated on RDG's site it might well be the same at other locations. 

[Paul Kelly]

On Sat, 3 Apr 2021 at 11:03, 'Michael Flynn' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

The Timetable Feeds on the Data Download page at RDG's http://data.atoc.org/ site aren't updating as per normal. I normally download the dataset on a Saturday for the latest schedule but last week the set (ttis951.zip), according to the header records, were updated to the 21st of March whereas I would have expected it to be to the Friday, i.e., the 26th and this week there is no new file available, the Current Timetable Feed is still last week's (27/03/2021, ttis951.zip).

Interesting observation - did you also notice the odd filename last week? Normally the counter on each file is offset by 7 from the previous week, but on 20 March it was ttis949 and then ttis951 on 27 March, i.e. only 2 days later, suggesting that something got stuck at the start of that week and hasn't been updated since.

 

I am aware that there has been or are other ways to access the data but I haven't kept a note of the details. I'd be grateful if someone was able to point me in the right direction, though I'm guessing if the data hasn't been updated on RDG's site it might well be the same at other locations. 

I think you can now get it from https://opendata.nationalrail.co.uk/ - at least there seems to be an option for it in the control panel; see attached screenshot.

--

BR Fares Ltd.
Work.Life, 33 King's Road, Reading, Berkshire RG1 3AR.
Registered in England & Wales No. 09074225.

[Michael Flynn]

Hi Paul,

Thanks your reply.  I did notice the sequence number hadn't advanced the normal seven from the previous week.  I just tried again but still just the 27th available.  I also notice now the latest Fares Feed is the 8th March.  

I think you might be right, it does look as though the schedule could be available from https://opendata.nationalrail.co.uk/.  Unfortunately, my password isn't working anymore again, I think as I haven't been active on the site for a while, and am now waiting for new one again ("If your email address is valid, you will receive an email with further instructions within the next 30 minutes.")

If still no joy I will send an email off to the RDG support team contact link.  

Cheers,

Mike

[Danny Withington]

I've noticed a few discrepancies with the RDG data recently. 

The fares data from March 1st was corrupt, so that may explain why there was a release for March 8th? 

If you don't mind using Network Rail's schedule data instead of the RDG's, Peter Hicks has a fantastic mirror of the CIF feed, available here: https://networkrail.opendata.opentraintimes.com/

[Michael Flynn]

Thanks, Danny. Don't mind if it comes from the Devil himself so long as the data is good ;)  

Schedule data does seem to be good except;

 - the RDG zip includes other files, most of which I don't use but one, the MSN file containing 'latest' TIPLOC data my processing does expect.  From memory, and obviously, the locations aren't going to be updated that often and I was able to use an existing RDG MSN file

 - my processing run threw out a lot of entries which I don't normally see, NP associations?  It's been literally years since I wrote the code but I remember and see from the output it's coded to skip any non JJ or VV associations.  There's hundreds of NP's!  Can anyone tell me what are these extra NP records?  Regardless, the data does seem good and I've now been able to update my sites.

Also, I remembered that https://opendata.nationalrail.co.uk/ deletes rather than suspends accounts after inactivity so after re-registering again and following links and instructions from the wiki I was able to generate a token and make a HTTP GET request for https://opendata.nationalrail.co.uk/api/staticfeeds/3.0/timetable.  Well, maybe!.. I used a Chrome add-on as the REST Client (not something I'm familiar with) and recieve a '200 OK'  But there is and continues to be green progress bars (over an hour now) but no other info and no obvious download.  I was thinking anyway, if it is the CIF schedule, what if you wanted different dates?  Who knows what this Timetable link is?  I'll let in run a bit more.  Won't matter anyway now I can use Peter's mirror for the latest CIF's until such time there's an update on the RDG site.

Cheers :)

[RailAleFan]

> There's hundreds of NP's!  Can anyone tell me what are these extra NP records?

I don't use the RDG timetable data but NP associations are "next working" in Darwin  - looks like the same codes.

Cheers

--

https://railalefan.co.uk/

[Michael Flynn]

>  NP associations are "next working"

I also see now from the CIF spec; 

"..  ‘JJ’ for Joining trains and ‘VV’ for Dividing trains. ‘NP’ for Next/Previous Associations may also be displayed but as this is an Operating association it should be ignored by journey planners. .."

Like I said it was years since I wrote my code for the CIF schedule and I just can't remember now whether it  pre-filters out these Next/Previous associations, or maybe they just don't list in the RDG data?  As it happens, I am committed to completely rewrite the code at some point, from PHP to Java, but I keep on having to put this back, and the last thing I want to do is have to revisit my old code (gives me headache even just thinking about it).  Maybe someone who doesn't get headaches about this stuff as readily as I do can tell me whether I am right to be skipping over (ignoring) these NP records.  What does that actually mean, anyway, next/previous association?  I think I thought about it before, wasn't quite sure, and decided just to go with the 'ignore'!?  Does it maybe mean it's for the train staff to know (operations as it says) but not relevant for passengers and thus timetables?  But I'm probably just going to wait till recode before having to think about it again anyway.  Would be interesting to know though..

 Anyway, glad to now have the up-to-date schedule and another source for the data and that's the main thing for me right now.  

Cheers again!

[Ian Sargent]

Next/Previous associations are for operating purposes only; showing that stock from an incoming train will be used to form another outging train. For passenger enquiries they should be ignored.

[Michael Flynn]

>> showing that stock from an incoming train will be used to form another outging train.

I thought as much though wasn't sure.  Useful for rail staff obviously.  Thanks for clearing up for me and confirming.

[Rail Delivery Group]

At some point data.atoc.org will be closed in favour of opendata.nationalrail.co.uk so I would definitely advise updating your code to use the data from that source (which is the same data used by ticket issuing systems and official journey planners - the data on data.atoc.org is an older format which is only kept going for this site (I believe).

On one of the other points mentioned, timetable data from RDG is processed so that it is not exactly the same as from Network Rail, although still within the CIF format. As with all CIF extracts from Network Rail, RDG chooses which records it wants within the extract, so it is possible to get other schedules from NR which aren't in the RDG feed (freight schedules are an obvious example of this). Additionally, we add retail service IDs (needed for reservations), align some of the schedules with association records (in terms of their date ranges) and a few other more subtle changes to make the data ready for retail systems and journey planners.

Just something for people to be aware of if they consider swapping between the two.

[Michael Flynn]

Hi RDG, 

Thanks for the update and explanation of how CIF data is aligned differently.  That would explain why I was seeing those extra NP records from the Network Rail data.

>> the data on data.atoc.org is an older format which is only kept going for this site (I believe)

My broad understanding is that timetable schedules are currently available in the CIF format and via Darwin Push Port in the form of an initial flat file and then update records as they happen.  Regarding the latter, I intend to use this data in time but haven't yet looked at it nor even know the format, is it CIF, or something else?   I'm sure I could glean more from the wiki though and will do so.

Also, I persevered with the https://opendata.nationalrail.co.uk/api/staticfeeds/3.0/timetable link and had greater success with Postman and get the zip which I see includes the familiar files, including for MSN, as per what I've come to expect from the data.atoc.org site.  But it is still the CIF data and I wonder, are you able to clarify whether the schedule in this format, i.e., CIF, is on it's way out, as it were, any time soon?  It's been discussed on this forum before, it is an OLD format, and I would expect an upgrade sooner rather later, but is it to go any time soon?  From the opendata.nationalrail.co.uk as well?  For me, I won't mind rewriting my code, it needs it, but I'd hope to be forewarned in good time.  I guess from your comments this is a forewarning, but do you have a timetable, for want of a better word, in mind for the total demise of the CIF? 

[Rail Delivery Group]

It starts life in Darwin as a CIF file from Network Rail and as part of its processing Darwin spits out the 48hr 'flat file' in a bespoke XML format, which strips out a lot of the information from CIF which is superfluous to Darwin and adds in data relevant to it (such as if a location has been cancelled). Not overly relevant to this thread but just for information Darwin does use and publish the NP associations and also contains empty rolling stock schedules.

The CIF format is a Network Rail owned format so we aren't in a position to comment on how long it will continue to be the main (only) format of official timetable data, other than to agree that it is ancient (as some on this thread will already know, it has its roots in punch cards!). RDG, like so many others, are a consumer of this feed and simply pass the data on in the same format it is recieved, allbeit supplemented with additional data as mentioned above.

I'm not aware or have heard of any plans from Network Rail to introduce a new format and to retire CIF so all I can really say is that it is still likely to be with us for many years to come. If anyone hears or knows anything different then please do post about it here!

[Michael Flynn]

Well I prefer the CIF to SOAP any day LOL 

Cheers  :)

[Phil Goodman]

Morning,

Grateful if anyone could provide any information they have on whether or not the file will be updated on http://data.atoc.org/data-download, please?

I am using this source of schedule data due to the enhanced RSID information (and also assume lack of freight data or 'non-advertised 'Q' trains), so not keen to look at switching to another source right now if at all possible.  

Have raised a couple of messages via the support link, but no response as of yet. 

Thanks

Phil

[Ian Sargent]

Have you seen the ATOC post above from 6th April?

[Phil Goodman]

Hi Ian,

Thanks - yes I have, but I am still not clear (perhaps missing something obvious!) - are you referring to this statement? 

At some point data.atoc.org will be closed in favour of opendata.nationalrail.co.uk so I would definitely advise updating your code to use the data from that source (which is the same data used by ticket issuing systems and official journey planners - the data on data.atoc.org is an older format which is only kept going for this site (I believe).

'At some point' implies to me that the existing ATOC site will continue to be updated, but perhaps am misinterpreting? Grateful if you/ATOC could confirm, please?

I am also not clear if the file from the proposed alternative (opendata.nationalrail.co.uk) site is in exactly the same format (ie with enhanced RSID information, no freight etc) - or is the same as the network rail data? If the latter, I will need to take account of these changes, as rely heavily on the RSID (which I believe can be reformulated based on a couple of fields in the network rail file, but it is not 100% accurate or necessarily updated correctly - please correct me if wrong!). Any information on how I can reliably get the RSID from the network rail data would be gratefully received. 

It is not a simple task to switch over to the proposed alternative source, so am hoping to get this information clarified by someone if possible, please?

Thanks

Phil

[Ian Sargent]

As it's the same data used by TIS and journey planners, the timetable feed will include the RSIDs.

Ian.

[Michael Flynn]

I based my code, as far as I can remember, on the spec linked to from the data.atoc.org site;

Timetable Information
Data Feed Interface Specification
Document Ref: RSPS5046
Version: 03-00

The link for the spec on the wiki regarding the https://opendata.nationalrail.co.uk/ is titled
”Timetable information data feed interface specification (RSPS5046 v.03-00)” but actually links to;

Timetable Information
Data Feed Interface Specification
RSP Document Ref: RSPS5046
Version: 01-00

Me, personally, I don't want to rake up my old code if I can help it.  I will start using the NRE data from this weekend.  If it doesn't work I know I can use the NR data from Peter's mirror site which, certainly on the face of it, does work for me.   I think the differences between the datasets are that there's are extra records rather than different records (but I'm really only guessing). My websites aren't 100% anyway (for example I don't report Rail Replacements at all, station details, train operators might be out of date, etc).  I I'm waiting till I rewrite before addressing these issues.

 For other use cases though I can totally see why accuracy could be more important.  

[Rail Delivery Group]

I think that's an error in the link and it should point to v03-00 as the title suggests. I've updated so it is now correct.

[Phil Goodman]

thanks all, sounds promising so will take a closer look

[Michael Flynn]

I've ran my CIF programs using the NRE (http://opendata.nationalrail.co.uk/) dataset, RJTTF972, with apparent success.  Apparent because I can't be certain which date the extract relates to; 

- the MSN file (RJTTF972) has comment records at the top of file, which includes a Generated value;

    /!! Start of file                                                               

    /!! Content type:  msnf                                                         

    /!! Sequence:      972                                                          

    /!! Generated:     06/04/2021                                                   

    /!! Exporter:      RjEhrTTT    

(The equivalent RDG (from http://data.atoc.org/) MSN file never had these header comments.)

- the Date of Extract field (specified as "Format ddmmyy defining the date that the BTD extract", Field 3, columns 23-28) in the header record of the MCA file, RJTTF972.MCA, is 190711?  In the RDG file this field was always set as expected, i.e., the Friday date before the Saturday download. (However, I now notice from the RDG, 27/03/2021 dataset, ttis951, the Date of Extract field is 020421?)

I am fairly certain the extract date could be inferred from the sequence number, i.e., the last 3 digits of the dataset (eg., 951 in ttis951.zip, 972 from RJTTF972.zip) but I'm only guessing and certainly can't be sure, it's not in the spec or on the wiki (as far I'm aware).  As I'll be using the NRE CIF dataset from now on, as advised, how can I be certain of the correct Date of Extract for it's MCA file and why is the value, apparently (I'm guessing it's not the timetables from 10 years ago!), not right ?

[Rail Delivery Group]

Having checked it looks like the dates are hardcoded in the full refresh files. No one seems absolutely sure why, or what the significant is of the 19/07/11 date. The dates are correctly set in the update files so whatever the reason it only affects the full refresh. I've made the relevant people aware within RDG so they can decide what to do about it.

I've also asked for this 'quirk' to be included in the next update to RSPS5046 as this should be captured there until, or unless, the hardcoding is removed and the dates are set correctly.

[Michael Flynn]

>> Having checked ..  ..I've also asked for this 'quirk' to be incl and the dates are set correctly

Thanks for this and for all your help and info.  I can now publish the schedules on my websites with confidence that the data is up to date.  I'll watch out for any updates and in the meantime set my Last Updated field manually.

[Phil Goodman]

FYI - the weekly timetable file on http://data.atoc.org/ is now updated and as I understand it, will continue to be updated - but the intention is to decommission at some point.

[Michael Flynn]

Thanks Phil, in that case, I'm just going to use timetable file from http://data.atoc.org/ for the time being.  Main reason, I just can't tell what date the data from the National Rail zip is!?  Maybe it could be inferred, but don't know how to anyway. 

[Michael Flynn]

Oh well, scrub that last comment.   Not sure where you're getting your information Phil but I've just been over there and the weekly timetable file on http://data.atoc.org/ is still showing as 27/03/2021.  

[Phil Goodman]

Definitely updating for us - latest is showing 17.04.2021 (on http://data.atoc.org/data-download).  

[mi...@a1publishing.com]

My mistake, I see it now. Thanks :)

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/9dbf5ca3-7b23-4217-a970-231665cc29f0n%40googlegroups.com.