Peter? Filtering RDM feeds for a single or small number of routes application — Darwin Push Port and TD

[Trevor Diamond]

Hello all,

I'm building a personal project around London Overground lines (the Lea Valley routes) 

I want to use the free feeds from National Rail as opposed to a third party.

At the moment, I'm consuming three feeds from the Rail Data Marketplace, all via Kafka:

- Darwin Real Time Train Information (Push) - the JSON Push
- Network Rail TD (TD_ALL_SIG_AREA)
- Network Rail Train Movements

Been playing with the Weaver Line and everything works well, and I'm getting genuinely good results - confirmed arrivals and departures, actual platform numbers, and real delay figures against the working timetable.

My question is about volume. All three topics appear to be full national firehoses, and I filter client-side down to 25 stations on one route. Measuring the Darwin topic over a six-minute window, I received 7,487 messages, of which 116 touched a Weaver station - about 1.5%. So I'm discarding roughly 98% of what I receive, continuously, 24 hours a day. TD looks similar in character.

That feels like the wrong shape, and I'd rather not be a needlessly heavy consumer of someone else's infrastructure. So....

1. Is any server-side filtering available on the RDM Kafka topics — by TOC, TIPLOC, or geographic area — or is client-side filtering simply the expected model?

2. For TD specifically, the legacy Network Rail STOMP feed offered per-area subscriptions rather than one combined topic. Is there an equivalent on RDM, or a route to requesting one? eg limiting to the Romford ROC areas covering West Anglia for Weaver

3. Is the "Darwin Real Time Train Information snapshots" API a better fit for a single-route application than streaming the Push Port - i.e. is it intended for exactly this case?

4. More broadly - is there a recommended pattern for small, single-route consumers? I may well be using these feeds in a way they weren't designed for.

Your steer on this much appreciated

Many thanks,

Trevor

[Peter Hicks]

Hi Trevor

On Sunday, 16 August 2026 at 21:12, Trevor Diamond <trevori...@gmail.com> wrote:

My question is about volume. All three topics appear to be full national firehoses, and I filter client-side down to 25 stations on one route. Measuring the Darwin topic over a six-minute window, I received 7,487 messages, of which 116 touched a Weaver station - about 1.5%. So I'm discarding roughly 98% of what I receive, continuously, 24 hours a day. TD looks similar in character.

That feels like the wrong shape, and I'd rather not be a needlessly heavy consumer of someone else's infrastructure. So....

1. Is any server-side filtering available on the RDM Kafka topics — by TOC, TIPLOC, or geographic area — or is client-side filtering simply the expected model?

I don't believe there is.

2. For TD specifically, the legacy Network Rail STOMP feed offered per-area subscriptions rather than one combined topic. Is there an equivalent on RDM, or a route to requesting one? eg limiting to the Romford ROC areas covering West Anglia for Weaver

The TD feeds are 'everything', and there isn't any filtering.  On the original NR Open Data platform, there was an attempt to align specific TDs with a geographical area, but this was updated in 2014 and not touched after that, so became pretty useless.

3. Is the "Darwin Real Time Train Information snapshots" API a better fit for a single-route application than streaming the Push Port - i.e. is it intended for exactly this case?

The snapshots API is an hourly snapshot of 'active' trains in Darwin and is intended to be used with the Push Port feeds to keep your local database up-to-date.  Whilst you could grab a snapshot each hour, it'd show the state of all trains across the country at that moment in time.

4. More broadly - is there a recommended pattern for small, single-route consumers? I may well be using these feeds in a way they weren't designed for.

The only recommendation is to do any filtering you think you need to do locally.  Trying to implement filtering centrally would mean a huge amount of complexity for a very small number of users.  Even then, the filters implemented might not be suited to everyone's needs.  In 2026, compute is a lot cheaper than it was 15 years ago, so you're best getting everything and filtering out what you don't want, rather than worrying about egress costs for data from the RDM.

Peter

[Trevor Diamond]

Makes sense - as always- I appreciate your steer!

T

--
You received this message because you are subscribed to a topic in the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openraildata-talk/qqW8NYnOPrM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/buPOYJJUMR6kGEciTm_7h06OEs8NokD2U4Q6GBQpItEY39TqK9LGDnswO01mgPbgni9_4wQMLIzZ2oWRXHnOawPcnckYk_cDY3ZhIgPNyeY%3D%40poggs.co.uk.