Outage

[Matthew Burdett]

As per grafana dashboard for public data feed, there is increased latency monitored and I am seeing outage on my Vstp and trust / TD feeds. 

Thought I'd put it out there. The notice at the top says it has been identified.

[Matthew Burdett]

I have logged a ticket as I am getting connection timeout. RDM movement feed hasn't updated since 5pm and TD Preview isn't loading at all.

Surprisingly I'm still getting TD messages from the RDM Kafka, but nothing through the NROD platform

Is anyone else experiencing the same?

Would be great if someone could please advise me how I can get access to approved data feeds instead, as a failover if possible. Thankyou 

[Jack Brewer]

I noticed the issue earlier, I’m fully back up and running on the public feed now. Interesting that I didn’t have any issues with the approved feed. Guessing that means it was a CACI issue with the public feed. 

Unfortunately I can’t help with getting on the approved feed. Have you tried contacting open...@networkrail.co.uk?

Jack

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/CA%2B2-wYH3fpOsGKAh4zkoi2Cui%3DxGd888p%3DcdLifHTABGttC4_A%40mail.gmail.com.

[Seb Dazeley]

I haven't looked in detail but my monitoring indicates I lost connection at 19:21 and reconnected at 19:51 but I haven't received any data from the TD or movement feed over the last hour since reconnecting. 

[Matthew Burdett]

Ok I'm using stomp and constant connection timeout. No problems with Kafka via RDM.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/b9ca2381-072a-4210-a42b-6ebb1510e091%40app.fastmail.com.

[Peter Hicks]

On Friday, 5 December 2025 at 20:43, Matthew Burdett <matthewbu...@gmail.com> wrote:

Would be great if someone could please advise me how I can get access to approved data feeds instead, as a failover if possible. Thankyou

Leave it with me - but it'll be Monday before I can do anything.

Please don't try to set up failover - the point of the 'approveddatafeeds' platform is to be a robust feed for production and 'visible' applications, and the times it fails when the public platform is up and running should be minimal or never.  There is more power in saying "the approved platform was down for 1 hour and these widely used and visible sites lost data" than "the approved platform was down for 1 hour but this site failed over to another platform and didn't lose quite as much data as everyone else".

Peter

[Seb Dazeley]

I've checked my logs and I actually also can't reconnect to NR/CACI. This is public feeds as well. It seems my monitoring thinks a 5xx error from a script that monitors the STOMP feed connection meant it was connected successfully... 

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/4QOM0hMYdcT1sZi0wN5bKfLJrqUcLIzVB6jXPzaB3Z6Mz6b5JhBX4PPVFMa1ng7pL2rR--gY0fh4A0JldexK7cTJZcBfGvOFAMeP1uoXs9M%3D%40poggs.co.uk.

[Matthew Burdett]

I'd appreciate the setup on that Peter, thanks. 

This is a bit of unfortunate timing, right after caci offices close for the week lol. I can't see many other websites affected other than Trainmapper app, so I'm guessing they're all on the approved platform or not using stomp 61628.

--

[Phil Wieland]

One of my clients, which uses publicdatafeeds, was disconnected at 18:14 yesterday (05/12/2025) and hasn't been able to get back on, it just gets connection timed out.  I'm guessing it has been firewalled due to too many connect attempts?  Does anyone know how long I need to leave it to get unblocked?

Phil

[Phil Wieland]

Sorry, should have said STOMP client on port 61618.

[Peter Hicks]

On Saturday, 6 December 2025 at 09:26, Phil Wieland <philw...@gmail.com> wrote:

One of my clients, which uses publicdatafeeds, was disconnected at 18:14 yesterday (05/12/2025) and hasn't been able to get back on, it just gets connection timed out. I'm guessing it has been firewalled due to too many connect attempts? Does anyone know how long I need to leave it to get unblocked?

Not official advice, but the rate-limiting strategy I proposed some years ago was to block a consumer from new connections a few minutes on more than a certain number of authentication failures in a period of time.  Blocking an IP address for too long just ends up being a pain in the backside, as appears to have happened here.  There's no value in punitive automatic blocking IP addresses either, or killing off existing connections.

I have a call with CACI and NR on Tuesday 16th and will raise this as an issue - as I've said before, any automatic block needs to have the policy clearly documented and have some kind of notification process around it.  A really good example is to have an error message sent back rather than silence.  HTTP and SMTP both do this.

But in the meantime, I'd suggest stopping the consumer for a little over 5 minutes, 15 minutes, 30 minutes, 1 hour, 3 hours, 6 hours and then 12 hours.

Peter

[Seb Dazeley]

I'm also still unable to connect.

My exponential backoff currently has a hard limit of five minutes - thanks for the info Peter. 

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/OsaugErnOeQrnZKTKZADwvbsqnkE2msDxYtoWaW89QnuVl_7ciiUC2C0fb8HUwt5chD37B96baE8B4P3rvoTehEiedlFroFWwVQH2Sf48_A%3D%40poggs.co.uk.

[Phil Wieland]

Many thanks for that, Peter, my maximum backoff is currently ten minutes, I guess I need to add some longer periods.

[Matthew Burdett]

Had my connection turned off since peters message 2 hrs ago, unfortunately no change for me. Must be a rigourous firewall setting their end.

Has anyone tried setting up from a new IP address? I've never experienced this before during outages. 

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/a7b076f5-aca6-43f9-a968-56fb87c5baf2%40app.fastmail.com.

[Christoper Stafford]

My development server has also failed to reconnect, even after an hour of being backed off. It does seem like there's more of a problem this time than there has been previously.

[Roy Doyle]

I've been happily using STOMP (publicdatafeeds.networkrail.co.uk:61618) for the last year or so, but now I'm just seeing connection timeouts. What's broken? I opened a support ticket, but I guess it's a Mon-Fri 9-5 response window.

[Phil Wieland]

It looks like the server is down, I can't connect to port 61618 even from servers which have never connected there before, so probably not a firewalling issue.

[Matthew Burdett]

Grafana shown dead Vstp since Friday

Now showing a burst of messages

I have connected successfully to STOMP 

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/24540410-722d-4c05-98f3-f80256921a9fn%40googlegroups.com.

[Phil Wieland]

Well spotted!  I'm back on.

[Roy Doyle]

Back on here too, phew!

[Seb Dazeley]

Back up here as well and getting data from both the TD and Movement feeds as usual. 

I see there is still no update on the Grafana status page since the claim that the issues were resolved, from Friday at 8:10pm.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/e024fac3-5fb5-4b13-865c-c2d3fcb8d391n%40googlegroups.com.