Short CACI NROD outages

[Alex Barber]

Hi All,

We are seeing regular outages at similar times (around 6pm) on the NROD feeds.

The last one was slightly longer -

2025/10/15 ~18:24 -> 18:29

This is visible on the grafana page - https://nrodcaci.grafana.net/public-dashboards/960fa54d94884dc7abd1f5ab9c70df7e

Are these known quantities? 

-Alex.

[Peter Hicks]

Hi Alex

I see these on the 'approveddatafeeds' platform too, so they're common to CACI.  However, I also see them on my own connectivity - between 18:31:25 and 18:33:51 and they're abnormal rather than graceful.

Since there's a peak of messages right after the outage, it suggests instant recovery, with message queues being messages queues and filling up with data that would otherwise have been lost.  However, that's for my connection which is pretty aggressive when it comes to reconnections, and CACI's systems may take a little longer to detect an outage and reconnect.

It could well be scheduled maintenance, or some item of network kit failing over and not maintaining state (e.g. firewalls).

When were the previous outages?  I'll check on my side to see if I see similar.

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/e7e6433e-50df-4849-bb85-075fd877e287n%40googlegroups.com.

[Alex Barber]

Here are some more examples. The outage on the 15th was significant as we seem to have totally lost all messages in that period. On other occasions it has only been partial. Note that times below are in UTC and my original message was using local time. for convenience I have added it on the end in UTC

2025-10-03 - 17:02 - 17:04 (approx. 5% of expected EL messages)

2025-10-04 - 17:25 - 17:27 (approx 25% of expected EL messages)

2025-10-10 - 17:01 - 17:03 (approx 33% of expected EL messages)

2025/10/15 ~17:24 -> 17:29

Times in UTC. EL messages = messages from signalling areas used by Elizabeth Line.

It is also interesting how the times seem to repeat.

[Peter Hicks]

On Thursday, 16 October 2025 at 11:07, Alex Barber <alexj...@gmail.com> wrote:

Here are some more examples. The outage on the 15th was significant as we seem to have totally lost all messages in that period. On other occasions it has only been partial. Note that times below are in UTC and my original message was using local time. for convenience I have added it on the end in UTC

2025-10-03 - 17:02 - 17:04 (approx. 5% of expected EL messages)

2025-10-04 - 17:25 - 17:27 (approx 25% of expected EL messages)

2025-10-10 - 17:01 - 17:03 (approx 33% of expected EL messages)

2025/10/15 ~17:24 -> 17:29

Times in UTC. EL messages = messages from signalling areas used by Elizabeth Line.

It is also interesting how the times seem to repeat.

Thanks - that sheds more light on the situation.  I can't see any similar outages on the 3rd, 4th or 10th, so I suspect the outage on the 15th was something different.

Your best course of action is probably to contact CACI in the first instance and report back what they say.  I anticipate they'll point at Network Rail and an answer might not be forthcoming - but please keep us posted, especially if you see further outages.

Best wishes,

Peter

[Alex Barber]

I have emailed them - I will post here if I get a response.

[Alex Barber]

See below - response from CACI

Dear Alex,

Thank you for bringing this to our attention.

We’ve reviewed the logs and can confirm that there was a brief upstream disconnect around the times you reported. These interruptions were very short-lived and did not trigger alerts from our monitoring systems, though we did observe a slight dip in message rate before normal service resumed.

As our data feed is provided by Network Rail, we’ve reached out to them to check whether any maintenance or other activity might explain these brief disruptions. We’ll keep you updated as soon as we receive further information.

Please don’t hesitate to reach out if you have any additional questions or information.

Kind regards,

CACI - Digital Solutions 

[Mark Rawlinson]

Just to show that these 'blips' can happen at any time of day, there was another one around 10.10 this morning.

[Peter Hicks]

On Friday, 17 October 2025 at 11:30, Mark Rawlinson <freightmas...@gmail.com> wrote:

Just to show that these 'blips' can happen at any time of day, there was another one around 10.10 this morning.

Having noticed that outage by having missing and stale data on one of the maps I was testing, I dug in further.

It turns out this blip wasn't something upstream at Network Rail, it was planned maintenance on the approveddatafeeds platform.  Did anyone get an email notifying them of this maintenance, or was it (again) only posted in the Latest Updates section of the Grafana pages?

Peter

[Mark Rawlinson]

Another one this evening (just after 1800 BST) - hundreds of messages lost as usual :-(

What would cause these outages to occur at such similar times (within a minute or so) on different days??

If it had been yesterday, I would have said a cron job set up to clear a cache or reboot a server every Friday evening,

but it's Saturday today!

MARK

On Thursday, October 16, 2025 at 11:07:05 AM UTC+1 Alex Barber wrote:

[Peter Hicks]

On Saturday, 18 October 2025 at 18:19, Mark Rawlinson <freightmas...@gmail.com> wrote:

Another one this evening (just after 1800 BST) - hundreds of messages lost as usual :-(

I see this on the public and approved platforms, and nothing on either of my direct connections.

I'll escalate this within Network Rail on Monday.

Peter

[Peter Hicks]

All,

FYI - this happened again at 1800:

The result?  Missing berth steps - 9G53 is not on the approach to Radlett, it's actually on the fast line south of Elstree and Borehamwood.

Peter

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/cZoZn67VHIJv1os2bGzd8uN7w-a_Rzatp9c_bX7Zj4u1uW6X99658gVWxHVW74uYPqGrlxfpGhkVR-FnzCLyrbtMOLTFPIFkq4hpNkYpAWU%3D%40poggs.co.uk.

[Mark Rawlinson]

Beat me to it!