CIF timetable files inconsistency

[Paul Clark]

Yesterday we detected 150 overlapping UIDs in the upcoming schedule and there seems to be disagreement between the daily CIF update files, the full timetable CIF and OpenTrainTimes.

e.g. 

There are 3 entries for W06039: https://www.opentraintimes.com/schedule/W06039/ 

(no overlaps on OpenTrainTimes)

The cumulative result of the CIF update files gives the same 3 entries but for the timetable starting 2025-12-15, the "valid to" date is set to 2026-05-15 instead of 2026-04-10. This causes it to overlap with the other timetables.

The full CIF file is missing the timetable entry starting 2025-12-15, so no overlaps.

I see this same pattern for the other UIDs.

Which is "correct"?

[Peter Hicks]

Hi Paul

On Friday, 20 March 2026 at 09:48, Paul Clark <paul....@esoterix.co.uk> wrote:

Yesterday we detected 150 overlapping UIDs in the upcoming schedule and there seems to be disagreement between the daily CIF update files, the full timetable CIF and OpenTrainTimes.

Just to check - are you using the actual CIF, as in the 80-column wide native format files from NROD?  Or are you using the JSON files?

Peter

[Paul Clark]

Hi Peter, we’re using the actual CIF files from NROD.

--
You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/zRSrf2mufZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/-G8oxmsfBCBqKQXF-bJR5M0-R-6hao8c1L-rDM1CJLHwEgje_sWoS5l68Hu8cpnBsi5D-uz7QLO2kDNjFK2HEFslONJ0fskxbKWHFclHU50%3D%40poggs.co.uk.

[Peter Hicks]

On Friday, 20 March 2026 at 10:07, Paul Clark <paul....@esoterix.co.uk> wrote:

Hi Peter, we’re using the actual CIF files from NROD.

OK, good - those should be unadulterated.

The first mention of W06039 (this year) was on 18th February - schedule valid SX (Monday - Friday) between 2025-12-15 to 2026-05-15.  Note that there may be entries prior to this date, but I'd have to pull files from the archive and I don't have time to do that right at the moment.

I then have three entries for W06039 which came in on 17th March:

• A revise for the existing WTT schedule starting 2025-12-15 to 2026-04-10

• A new overlay starting 2026-04-07 to 2026-04-10 valid MSX (Tuesday - Friday)

• A new WTT schedule starting 2026-04-13 to 2026-05-15 valid SX (Monday - Friday)

Do you see the same records in the CIF files that you have?

Peter

[Paul Clark]

I don’t have the revise of the schedule from 2025-12-15 to 2026-04-10. I’ve checked all the way back to September 2025 when this timetable was created.

 

Do you have a date for when you got the revision?

--

You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/zRSrf2mufZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/ZfLUlL92CdieI_0JQex4hoCtUusTLKE7coQMVxN30Fib8VRq1UZ3roLiHmIMYiXT11A6PlIBALt7h6tIVgrcn4X6ekhDoQ6Y00piROZx9Hc%3D%40poggs.co.uk.

[Peter Hicks]

On Friday, 20 March 2026 at 10:28, Paul Clark <paul....@esoterix.co.uk> wrote:

I don’t have the revise of the schedule from 2025-12-15 to 2026-04-10. I’ve checked all the way back to September 2025 when this timetable was created.

 

Do you have a date for when you got the revision?

That was in the CIF update dated 17th March 2026.

Peter

[Paul Clark]

I’d forgotten that we have made a change to get our CIF updates from RDM, instead of Network Rail.

 

The issue appears to be that some of the updates were missing from the RDM CIF file (see attached). They also don’t appear in adjacent RDM files.

 

Thanks for the help.

 

From: 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com>
Sent: 20 March 2026 10:47
To: openrail...@googlegroups.com
Subject: RE: [openraildata-talk] CIF timetable files inconsistency

 

 

 

On Friday, 20 March 2026 at 10:28, Paul Clark <paul....@esoterix.co.uk> wrote:

--

You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/zRSrf2mufZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/LQ7J1s6qB9MDdzh-WjSK0GbTSpYMiLQaVv2wWi6QZXKe1n3vqEhMJfJ0kfPRfjWLL3s-PKvMPidJMmh9_tzRTHBiSYlrJ2khWkxBai2uKzc%3D%40poggs.co.uk.

[Peter Hicks]

Hi Paul

On Friday, 20 March 2026 at 12:10, Paul Clark <paul....@esoterix.co.uk> wrote:

I’d forgotten that we have made a change to get our CIF updates from RDM, instead of Network Rail.

 

The issue appears to be that some of the updates were missing from the RDM CIF file (see attached). They also don’t appear in adjacent RDM files.

That would explain it - one of the schedules is a 'runs as required' (characteristic Q) and therefore isn't included in the CIF from RDG.  It only contains passenger services.

If you want to use that CIF, clear down your database and import a full extract, otherwise you'll be lacking updates to any non-passenger train!

Peter

[David Wheatley]

It's only tangentially related, but the filtering of services in the RDM CIF from RDG means that most people would need to make changes to their timetable importer system. Not only do you miss out on updates for services like these, but you also miss out on services which change between non-passenger/unadvertised and passenger-services.

As an example, the 14:40 KGX - HGT service on 30 March was amended in the NROD CIF to be unadvertised, but if you only consume the RDM timetable you'd still think it was a publicly advertised service as the overlay to make it non-passenger is stripped by RDG's DTD service. As a result, retailers will still sell tickets for the train that quite likely won't even run.

Similarly, if an operator had an unadvertised train in the LTP and STP overlaid it to make it a passenger service, you could end up with an overlay schedule for a LTP schedule that doesn't exist as that would have been stripped from the CIF.

If you do actually want an accurate timetable, even if you only care about passenger services, you should always use the NROD timetable. And if you want RSIDs, you'd probably be better off mangling the two together at import-time to extract that data.

David

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/dOrQjAoaaYIdEA_iv-UN997ZLgqQ_Lyh1kTPVyZUyskaqVB0qzjlrXUA3o-Y4HrgOTf0q35DQwdEyFEWQbf2NCdYvDQXOPwGSP4bkEVnBlU%3D%40poggs.co.uk.

[Ian Sargent]

Really bad practice to amend the category of an Advertised train in an overlay to be Unadvertised. The scheduled should have been STP Cancelled, and then a new Unadvertised schedule inserted under a different UID.

[Paul Clark]

Thanks, that all makes sense now.

 

Do you know of somewhere I can get the last few months of network rail CIF update files? I need back until 21^st January.

 

From: 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com>
Sent: 20 March 2026 12:13
To: openrail...@googlegroups.com
Subject: RE: [openraildata-talk] CIF timetable files inconsistency

 

Hi Paul

--

You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/zRSrf2mufZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/dOrQjAoaaYIdEA_iv-UN997ZLgqQ_Lyh1kTPVyZUyskaqVB0qzjlrXUA3o-Y4HrgOTf0q35DQwdEyFEWQbf2NCdYvDQXOPwGSP4bkEVnBlU%3D%40poggs.co.uk.

[Tom Cairns]

It may be considered bad practice but I don’t see why it is bad practice (is it because the RDG CIF acts in strange ways around Q/Y/OU/XU?).

 

It seems eminently sensible if you’re adjusting a train to go from unadvertised to advertised (or vice versa) that you just need to alter the category.

 

It’s entirely possible on the RSP CIF to get cancellation records for a permanent schedule you have no record of! The Network Rail version of CIF is intact and … fine.

 

We (in RTT) alert TOCs as and when we see these issues which tend to come in waves, but on average we see a couple of services a month impacted by this.

 

As David intimates, the only thing you’re really gaining from the RDG CIF is the RSIDs which are only particularly relevant in a retail context in any case.

 

 

Tom

 

 

From: openrail...@googlegroups.com <openrail...@googlegroups.com> On Behalf Of Ian Sargent
Sent: 20 March 2026 15:23
To: A gathering place for the Open Rail Data community <openrail...@googlegroups.com>
Subject: Re: [openraildata-talk] CIF timetable files inconsistency

 

Really bad practice to amend the category of an Advertised train in an overlay to be Unadvertised. The scheduled should have been STP Cancelled, and then a new Unadvertised schedule inserted under a different UID.

On Friday, 20 March 2026 at 12:24:52 UTC David Wheatley wrote:

It's only tangentially related, but the filtering of services in the RDM CIF from RDG means that most people would need to make changes to their timetable importer system. Not only do you miss out on updates for services like these, but you also miss out on services which change between non-passenger/unadvertised and passenger-services.

 

As an example, the 14:40 KGX - HGT service on 30 March was amended in the NROD CIF to be unadvertised, but if you only consume the RDM timetable you'd still think it was a publicly advertised service as the overlay to make it non-passenger is stripped by RDG's DTD service. As a result, retailers will still sell tickets for the train that quite likely won't even run.

 

Similarly, if an operator had an unadvertised train in the LTP and STP overlaid it to make it a passenger service, you could end up with an overlay schedule for a LTP schedule that doesn't exist as that would have been stripped from the CIF.

 

If you do actually want an accurate timetable, even if you only care about passenger services, you should always use the NROD timetable. And if you want RSIDs, you'd probably be better off mangling the two together at import-time to extract that data.

 

 

David

 

On Fri, 20 Mar 2026 at 12:13, 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

Hi Paul

 

On Friday, 20 March 2026 at 12:10, Paul Clark <paul....@esoterix.co.uk> wrote:

I’d forgotten that we have made a change to get our CIF updates from RDM, instead of Network Rail.

 

The issue appears to be that some of the updates were missing from the RDM CIF file (see attached). They also don’t appear in adjacent RDM files.

 

That would explain it - one of the schedules is a 'runs as required' (characteristic Q) and therefore isn't included in the CIF from RDG.  It only contains passenger services.

 

If you want to use that CIF, clear down your database and import a full extract, otherwise you'll be lacking updates to any non-passenger train!

 

 

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/dOrQjAoaaYIdEA_iv-UN997ZLgqQ_Lyh1kTPVyZUyskaqVB0qzjlrXUA3o-Y4HrgOTf0q35DQwdEyFEWQbf2NCdYvDQXOPwGSP4bkEVnBlU%3D%40poggs.co.uk.

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/dc91312d-ccbb-4ec5-b28a-90566fcd96c0n%40googlegroups.com.

[Peter Hicks]

On Monday, 23 March 2026 at 10:37, 'Tom Cairns' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

It may be considered bad practice but I don’t see why it is bad practice (is it because the RDG CIF acts in strange ways around Q/Y/OU/XU?).

 

It seems eminently sensible if you’re adjusting a train to go from unadvertised to advertised (or vice versa) that you just need to alter the category.

I have a feeling the RDG CIF is not processed by the same codebase as the CIF generated from ITPS data.  There are some interesting cases around schedule revisions when your CIF profile has filters applied - such as when you have a schedule revision for a schedule that is 'out of scope' for your user, but that schedule is then revised to be in-scope by the addition of extra timing points.  A geographically filtered user should ideally receive the revised schedule as a new (as it's new to that user), and if the schedule is then revised and becomes out-of-scope, the user should receive a delete.

I have only ever consumed an un-filtered CIF with everything in it, so don't have first-hand experience.

Peter

[Ian Sargent]

There are hundreds of data feeds that use the CIF data, and each is specific to the needs of the recipient system. Some feeds take every single schedule, whilst others are filtered to provide a sub-set of data. For example: freight only services, passenger only services, trains operated by a particular TOC, trains passing through a particular signalling area, trains calling at a single station.

Many of these only require data for certain categories of trains. For retail systems, only advertised passenger trains are required, which means that schedules in the OU and XU categories are not included when Network Rail compiles that particular data feed. Hence any change to a train's category that swaps between advertised and non-advertised categories causes problems and is bad practice. In such circumstances, the original train should be STP Cancelled for the date/s in question and a separate schedule (using the new category) inserted.

[Oscar Harris]

Hi Ian,

Fundamentally, I see two options here for this filtering:

1. Make the filtering less stateless - there's no reason the code doing the filtering couldn't be aware of the existing state of the services, and therefore produce a correct and "valid" filtered CIF: don't drop overlays that make an advertised service unadvertised on the floor, don't include an overlay for a service with no permanent schedule, etc.
2. Decide that the above is hard, and therefore changes that would make a schedule change from filtered to unfiltered or vice versa are considered wrong as you suggest. If you go down this route, then changes doing this should be simply rejected. If you are doing this filtering downstream of the canonical source, and cannot get upstream to enforce these invariants, then option 1 is your only choice.

I don't think simply mangling the data into a format where the referential integrity is kaput and with issues that cause passengers to be missold tickets for trains that have been edited to unadvertised to be an acceptable answer, and we can't just say it's bad practise when it happens. Either it's allowed and the system should cope with that, or it isn't allowed and the changes should be actively protected again.

(incidentally I'm not a particular fan of option 2, it can be a real pain when services are STP C and N'd when they're the same service because ideally you need a coherent history of a service for things like customer messaging - it's bad to think a service is cancelled when it's actually just changed to be under a new UID. So encouraging it wouldn't be ideal)

Oscar

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/b3e12ada-8f42-4de3-b7c6-8d35c120dda1n%40googlegroups.com.

[Ian Sargent]

Well, unless someone can come up with a budget to amend the software for 1, we are left with 2. 

When I first came into contact with CIF data in the early 1990s, at the six train planning units around the country there was quite a discipline in place about how services should be amended. In part this was required by the constraints of system they were using at the time (TSDB). 

When that old system was replaced by ITPS at Network Rail it was followed shortly after by the three remaining train planning units being concentrated at Milton Keynes, with a lot of the old hands taking early retirement. That loss of expertise did, in my opinion, lead to certain liberties being taken with train planning that can have unforseen effects on the downstream systems that receive the data. 

[Tom Cairns]

What has come before doesn’t mean that we should allow or build in daft human-led restrictions now.

 

To pick up that Harrogate example, it shouldn’t really need a brand new identical path under a new UID to be loaded just to make it unadvertised – albeit yes, I know CIF requires the full LO/LI/LT entries for an overlay to make it XU. But for the Network Rail CIF it allows us to see that it’s still running on that day without having to investigate each day in detail; I don’t see why the RDG CIF can’t just mark that as a CAN rather than just dropping the schedule. Equally an Q/Y/XU becoming an XX, it should be an STP (N) rather than a VAR (O) and then getting all the CAN entries for the base WTT schedule I will never see. I won’t pretend to know where the filtering happens at either end but this really doesn’t seem a particularly hard problem to solve to improve the integrity of everyone’s database.

 

So on that basis I’ll throw my hat in the ring to fix it for free… (don’t worry I know that it won’t be taken up as it won’t be a gold plated solution by a big system integrator.)

 

On the flip side, while I’m not a huge fan of AI and LLMs*, I suspect that this is such a small, well-defined, problem that anyone with some domain knowledge could fix this problem for peanuts using that with sufficient testing.

 

Tom

 

* I would however note that I’ve just been working on a personal project to improve a solution to a very complex problem (in a rail context) over the last 3 weeks with Claude Code purely out of curiosity whether it could do it and it’s proved itself to be surprisingly capable. Add sufficient guard rails and testing in and it’s fine. The human equivalent to the aforementioned problem took a development team over a year in human-hours to solve. This problem is child’s play compared to that.

 

From: openrail...@googlegroups.com <openrail...@googlegroups.com> On Behalf Of Ian Sargent
Sent: 23 March 2026 15:30
To: A gathering place for the Open Rail Data community <openrail...@googlegroups.com>
Subject: Re: [openraildata-talk] CIF timetable files inconsistency

 

Well, unless someone can come up with a budget to amend the software for 1, we are left with 2. 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/10449919-2c83-421b-b200-e4079ceb1b22n%40googlegroups.com.