NRDP -> RDM for Darwin Push Port feeds

[Peter Hicks]

All,

I've just been made aware that there are plans to migrate Open Data users of the Darwin Push Port over to the RDM some time in early 2026.  Do not panic :)

Now is the time to start looking at the RDM version of the Push Port feed if you're currently using opendata.nationalrail.co.uk, and this group is of course available to discuss any problems you find.

Peter

[Evelyn Snow]

Hi Peter,

Know at all if AMQP and STOMP will be options by then?

Evelyn

[Peter Hicks]

On Tuesday, 3 June 2025 at 11:54, Evelyn Snow <eve...@kanaya.dev> wrote:

> Know at all if AMQP and STOMP will be options by then?

50% of me hopes it's 'yes'. 50% of me is slightly concerned that it'll only be Kafka as it's a Confluent-hosted service. This is something I've have raised this on multiple occasions in the past.

If it looks like it will be Kafka-only, I'll be quick to put out code and/or instructions on building a Kafka to ActiveMQ bridge for anyone who needs time to learn Kafka's way of doing things, or write a client for Haiku[1]. However, for the majority of people, I don't think this will be an issue.


Peter

[1] https://xkcd.com/806/

[Jez Smith]

Evelyn and all,

RDM currently only has Kafka.  We did a bit of a straw-poll early on in the project and Kafka was the preferred choice by those that responded.  We also looked at it technically, and Kafka was judged less likely to give performance problems.

We don't have any plans to make it available in MQ, but if there was significant demand we would reconsider that decision, however it would be jostling for space in a very limited change budget with the rest of the suggested RDM improvements.

I could be bias(!), but RDM is a robust scalable platform with support in place, and contains many other data products besides.  The long-term direction of travel (DfT, GBR etc) is to bring together and offer access to all 'non-PII' data on RDM. 

If you do subscribe to the Darwin feed, you can access it in JSON, AVRO, and XML.

Cheers

Jez

[Evelyn Snow]

Hi Jez,

For the Darwin push port, I think it's less of an issue considering it's already quite a good test
for non-compliant STOMP clients, but for other real-time feeds, I worry that this is likely to make
use by hobbyists that little bit harder. It's a fairly marginal problem, in fairness.

In my opinion, the more serious issue with the RDM for Darwin is the file transfer mechanism, a
regression compared to the "pull" option we currently have with the NRDP.

The RDM has potential, but it's full of gaps where the reasonable expectation a technical user
would have for day one functionality still isn't met. File transfer came late and still isn't fit
for purpose, but neither can the platform remember that I'm logged in if I open a new tab.

Not something I have an opinion on either way, but will the non-RDM LDB offerings be retired? Would
be useful info for the NROD wiki.

Evelyn

> --
> You received this message because you are subscribed to the Google Groups "A
> gathering place for the Open Rail Data community" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to openraildata-t...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/
> openraildata-talk/2948743a-0f30-4de6-8d00-1025877d351an%40googlegroups.com.

[David Wheatley]

My understanding is that LDB outside of RDM is already closed to new sign-ups. It's inevitable it'll be closed in favour of the RDM proxies (which are likely more friendly to new people, given it's JSON, and also seemingly no rate limits!).

The most frustrating part of LDB on RDM is how lots of endpoints are hidden under multiple different "products" and you need separate credentials for each. I ended up subscribing to maybe 5 products before I found the one with the actual endpoint I wanted. It'd be much better with one LDB and one LDBSV product with one set of credentials for each. I'd love to understand the reasoning for how this was initially set up.

For those using Huxley2 (which is what I used to use for LDB on personal projects), the structure is largely identical on RDM minus some enums being strings versus numeric values in Huxley.

Around the time someone else requested access and was refused (I think it was mentioned on here), I managed to get a new token approved but I believe that was solely as the request came from a retailer. That took a few days because their system for creating new tokens was broken at the time apparently, too!

David

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/aECU45MeRnhEtE%2Bi%40evelyn.moe.

[Peter Hicks]

Hi Evelyn

On Wednesday, 4 June 2025 at 19:48, Evelyn Snow <eve...@kanaya.dev> wrote:

> In my opinion, the more serious issue with the RDM for Darwin is the file transfer mechanism, a
> regression compared to the "pull" option we currently have with the NRDP.

I agree with you here - if there's not a way for software to respond to the "Timetable available" message from Darwin with a "Get the timetable file from this URL", then it's going to be difficult for consumers to ensure they're consuming up-to-date data.

A simple solution could (other solutions may be available) be to have an API through the RDM where you can call a URL with the filename in the Darwin XML message, and the API returns the file from the S3 bucket. That would avoid having to have a 'push' model and sounds like it'd fit nicely within the RDM architecture.


Peter

[Chris Bailiss]

Thanks Peter for the advanced notice of this upcoming change.

This page describes the Darwin feed on RDM as "broken" but with no more details:

https://wiki.openraildata.com/index.php/Rail_Data_Marketplace/Feeds

Is this an old/outdated note or is it still "broken"?  If yes, in what way(s)?

Is the shift from NRDP to RDM solely a change in distribution/transport mechanism and the relevant XML schemas are the same?  Or is RDM running/publishing an updated version of the Darwin data?

If there are schema changes involved, are these documented anywhere?

I tried following the "Darwin Real Time Train Information" link on the wiki page above but got a "Something went wrong" error (see attached).  

Is the link broken?  Or is this a poor mans "You need to be logged in to see this" message?

I have filled in the registration form but await that being processed to get a login.

Lastly, with this change upcoming for Darwin, is there a similar shift planned for NROD TRUST/Schedule/... feeds etc? 

Thanks

Chris

[Evelyn Snow]

That's an outdated note, and the product has since been replaced, but the list of
feeds on the wiki hasn't been updated.

https://raildata.org.uk/dashboard/dataProduct/P-d3bf124c-1058-4040-8a62-87181a877d59/overview

The RDM's Darwin is v18, schema here:
https://www.rspaccreditation.org/publicDocumentation.php#RSPS5051

Evelyn

> --
> You received this message because you are subscribed to the Google Groups "A
> gathering place for the Open Rail Data community" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to openraildata-t...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/

> openraildata-talk/7aa9f588-e6e0-447f-b753-64def524f3e2n%40googlegroups.com.