Network Rail Open Data Outage 2302-2308

[Peter Hicks]

All,

Did anyone else see an outage to the NR Open Data feeds from just after 2300 to just before 2310?  The message graphs on my infrastructure here (two separate servers) show a sharp dropoff in messages.

Peter

[Peter Mount]

There did look like there was a dip around 11.

Hopefully the image is attached but from

http://status.area51.onl/

I saw the blip in the last hour on both trust & td feeds.

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Bailiss]

Yes, I saw that too.

Chris

[Peter Hicks]

On Sat, 7 Nov 2015 at 00:12 Chris Bailiss <cbai...@gmail.com> wrote:

Yes, I saw that too.

@Chris, @Peter - did you see the outages/lumpy bits between 2200 and 2300 last night too?  (note: the peak between 0300 and 0600 is Darwin schedule messages which are normal)

 

@All - what impact does an outage or message loss have on your applications?

Peter

[Peter Mount]

Not as dramatic like that but there was an obvious blip.

That said with darwin the usual 2am restart took longer this morning. Even the ftp site was down for almost 2 hours (to me anyhow). The push feed about the same.

[James Singleton]

Yes, I saw a lower than usual amount of messages just after 2300 on the 6th. This was followed by a higher than usual amount of messages just after which suggests a temporary connection issue. The messages probably backed up in a queue and then got flushed when the connection was restored.

Today Darwin seems to not be providing any useful data since about midday. Including the live departure boards on the national rail website not showing any information. The disruption emails say this is due to Trust and TD being unavailable but my TD and train movement feeds have been ticking along as normal all day. This suggests that it isn't a problem at Network Rail's end and it's only National Rail Enquiries who can't connect.

Has anyone noticed any issues with feeds or with live departure boards at stations today?

James

[petermount]

Other than the one you've noticed, from 12:15 - which some toc's are also mentioning now - Southeastern did about 20 mins ago.

I'd suspect NRE will connect differently to how we do, which would explain why we're still getting Trust & TD but they arent.

Peter

[Peter Hicks]

Hi James

> On 10 Nov 2015, at 15:47, James Singleton <jpsin...@gmail.com> wrote:
>
> Today Darwin seems to not be providing any useful data since about midday. Including the live departure boards on the national rail website not showing any information. The disruption emails say this is due to Trust and TD being unavailable but my TD and train movement feeds have been ticking along as normal all day. This suggests that it isn't a problem at Network Rail's end and it's only National Rail Enquiries who can't connect.
>
> Has anyone noticed any issues with feeds or with live departure boards at stations today?

It looks like the connection from Network Rail in to Darwin is having difficulties, because Open Data is fine.

There are numerous failure modes which can use Darwin not to receive train movement data, and this is one of them. I suspect it’s a busy day for somebody out there…!


Peter

[James Singleton]

Hi Peter(s),

I suspected as much. The emails suggest that the problem is at Network Rail's end but ATOC probably have a dedicated connection and that's what has failed. I wonder if the physical departure boards at stations are still working.

I saw a complete outage on the Darwin Push Port at 12:15 which resumed back to normal at 12:45. At 13:00 the message rate dropped to a very low level (but not to zero like the previous brief outage) and continues low as of now.
I recorded a surge of messages after the initial outage which suggests a temporary connection issue to Darwin and catching up. I also logged a surge before the first outage (at 12:00) which is strange. All was normal up until then.

James

[Peter Hicks]

On 10 Nov 2015, at 16:19, James Singleton <jpsin...@gmail.com> wrote:

> I suspected as much. The emails suggest that the problem is at Network Rail's end but ATOC probably have a dedicated connection and that's what has failed. I wonder if the physical departure boards at stations are still working.

It could be anywhere - unwise to point fingers, especially if anything network-related exists.

Yes, departure boards at stations are still working. Most CIS has a ‘fallback mode’ where it will display local information and predictions, if it can. Crewe’s CIS earlier today was showing Delayed on all summary screens, and the platform displays were blank.


Peter

[James Singleton]

On Tuesday, 10 November 2015 16:23:19 UTC, Peter Hicks wrote:

On 10 Nov 2015, at 16:19, James Singleton <jpsin...@gmail.com> wrote:

> I suspected as much. The emails suggest that the problem is at Network Rail's end but ATOC probably have a dedicated connection and that's what has failed. I wonder if the physical departure boards at stations are still working.

It could be anywhere - unwise to point fingers, especially if anything network-related exists.

Agreed. The latest status email says: "The Trust and TD feeds are not available resulting in train movements not being reported. Network Rail technical engineers are currently working on the issue, no eta for resolution is currently available." and "The Network Rail Trust/TD feed is down".

It would be interesting to see a post mortem and I hope it is made public. Transparency is very good for building trust.