Contact for Open Access Data

[Tom Early]

Who do I contact for Darwin Push Port (STOMP) access?

[Peter Hicks]

Hi Tom

On Wednesday, 21 January 2026 at 18:29, Tom Early <tom....@gmail.com> wrote:

Who do I contact for Darwin Push Port (STOMP) access? 

You can access this via opendata.nationalrail.co.uk, or (preferred) the Rail Data Marketplace.  Both are self-service signup.

Peter

[Tom Early]

Hi,

Thanks for that. I have signed up for RDM

Sent from Outlook for iOS

---

From: 'Peter Hicks' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com>
Sent: Wednesday, January 21, 2026 6:30:48 PM
To: openrail...@googlegroups.com <openrail...@googlegroups.com>
Subject: Re: [openraildata-talk] Contact for Open Access Data

 

--
You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/RkDWTMgMlxA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/vGp2z2s6UwJFyj0aGJhIPOB8SruNOMf_4VNiqDT-zrI_1ZIxuPPqNvXn6HP729C0MRBMPyOMTcV2pL7Sc40ys9nU-YDnrQWrzh1krYZkY-A%3D%40poggs.co.uk.

[David Wheatley]

Hi Tom,

Be aware that the RDM feed has some fundamental issues regarding the order you receive messages, which may result in you storing the "incorrect" state of services if messages are sent on the feed for the same service in short succession.

https://groups.google.com/g/openraildata-talk/c/SU3R7k9Kwks

https://groups.google.com/g/openraildata-talk/c/oth6M9vnsKU/m/rHtISTmyAwAJ

https://raildata.org.uk/community/category/topic/postDetail/94

If you can set up your system to be connection-agnostic, so that you can use STOMP now and swap it out for RDM's Kafka in the future when these issues are resolved, this would be my suggestion.

David

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/AS8PR07MB7192CC41C76AD77FC10FACAEFB96A%40AS8PR07MB7192.eurprd07.prod.outlook.com.

[Lawrence Job]

Hi David,

Have you had an affirmative reply from RDM that they're going to address this? I was consuming the queue with a buffer to reunite them but now they're hovering around 5,000-20,000 messages out of sync which takes it from annoying to impractical to work with, even with a buffer, because buffering means lagging substantially behind realtime.

My understanding is that we have to stop using STOMP by March? I hope I misunderstood?

Kind regards,

Lawrence Job

[Jez Smith]

Lawrence,

To try to address some of your questions.

Firstly, yes, RDM is planning to address some of the issues relating to those data sources, specifically we are planning a snapshot schema version.  This will still remain a kafka topic based upon the push port schema v18.  We are planning to have this in place before the end of March.

You also asked about the existing NRDP access which I understand is being extended beyond March to recognise the tight timescales for any transition activities.  I do not know how long that might be extended for but the assumption could be made that this will be a number of months, rather than a number of weeks.

I hope that this helps.

Cheers

Jez

[David Wheatley]

Hi Jez,

> RDM is planning to address some of the issues relating to those data sources, specifically we are planning a snapshot schema version.

When you say "some of the issues", does this include the partitioning issue? It would be useful to know for certain in order to plan migrations.

Thanks,

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/adfa0330-bc69-4cb0-b8d4-d1c22f4bda1cn%40googlegroups.com.

[Jez Smith]

Yes, we are also looking at message loss because of partition issues.  Problem statement below.

Kafka topics for the Darwin Real Time Train Information pub/sub data product have two partitions. Some consumers are losing messages because certain messages exist only on one partition, and keys may direct messages to different partitions. Message loss may occur if consumers are misconfigured to only one topic/partition.

1. Identify configurations that may cause message loss.

2. Ensure consumers can read from all partitions for a topic.

3. Provide configuration guidance to prevent missed messages.

This is all beyond my understanding, so grateful if any responses used single syllable words!

Jez

[David Wheatley]

Thanks Jez, that's great to hear.

For what it's worth, the issue isn't necessarily that there's two (or more) partitions, but that messages about specific services aren't consistently sent to one or the other.

Each partition seemingly has its own different delay behind the actual open STOMP feed, and the order of messages received is only supposed to be in the correct order within each partition.

If, on a normal Kafka feed, you receive:

- message 1 on partition A

- message 2 on partition B

- message 3 on partition A

You can know that messages 1 and 3 will have been received in the correct order, but you don't know whether message 2 comes from before or after messages 1 and 3.

The result is that you might be told that a stop on a train is cancelled on one partition, and the same stop is delayed on another partition, you don't know what order the messages were intended to be received in.

If any production app was relying on the RDM Darwin feed, it would likely tell passengers fundamentally incorrect information about a train. It could send out notifications about stops being reinstated when they are actually cancelled, or notifications about stop cancellations when they have been reinstated. The issues make the data feed *impossible* to use for anyone who actually wants to provide accurate real-time information.

If an industry system (e.g. a TOC delay repay portal) was using this feed instead of the internal feed, it would end up wrongfully paying out compensation (thinking delays or cancellations happened when they were reversed), or denying passengers their compensation (wrongfully thinking cancellations or delays were reversed when they weren't). I hope you'd agree that isn't ideal!

The solution is to configure message partitions so that every message about a specific service or station (e.g., for station alerts) or health checks are always sent to a consistent partition. That could be as simple as partition 1 for an even UID and 2 for an odd UID, or a more complex algorithm.

Fixing this is all well and good, but Peter highlighted an even bigger issue a few weeks ago that messages aren't even in the correct order within an individual partition, which means that even if this is solved, there may well still be further problems to resolve before migrating off of NRDP is feasible. I'm not sure if the problem statement covers that issue, so it is worth highlighting again here: https://groups.google.com/g/openraildata-talk/c/SU3R7k9Kwks

I do appreciate that RDM is trying to fix these issues, but I hope you can also understand our frustrations when the new open data platform has these problems which aren't present in the original data feeds, while people are also pushing others to migrate to RDM without highlighting the potential consequences! 

Cheers,

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/d355a8bf-f050-4db1-aff6-45e1d59e681cn%40googlegroups.com.

[DCJ NaveX]

Hello,

 

We plan to reboot the server tonight at around 23:00. We will monitor our systems and take any action required if needed.

 

Regards,

Dereck Jones

I.T. Director.

NaveX Systems

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/ef42f023-5dbe-46cd-b6bf-c969f3d3e91an%40googlegroups.com.

Virus-free.www.avg.com

[Jez Smith]

Thanks David for your helpful comments which have certainly clarified the issue for me.

I have passed those comments on to the development team, so hopefully they can find a way to resolve them.

Cheers

Jez