NROD Approved outage from 1100

[Peter Hicks]

Afternoon

Hands up anyone else who has seen an outage on the 'approved' datafeeds platform from 1100 until now.

Peter

[Katie Weston]

I have heard people saying Traksy and RTT were down from 11 so sounds likely 

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/CAKGkhzaRKD0--orD%2B1PcfXh%2B8Rw2gYrHq_RthVgW9tx9eWd6tg%40mail.gmail.com.

[Mark Rawlinson]

RTT seems to be working again - presumably they have temporarily switched over to one of the older data feeds??

MARK

[Peter Hicks]

I think so - I've just given up and done so.

4h outage and counting.

Peter

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/cc5eb80f-f7f1-4997-a219-15cad45fd7e4n%40googlegroups.com.

[QRail Comms]

We are seeing the same

It looks like an odd failure. Data coming through a few messages at a time every 10 min

And the backlog/latency is more than 10 hours and counting.

[Peter Hicks (Poggs)]

> On 15 Jul 2023, at 21:07, QRail Comms <qrail...@gmail.com> wrote:
>
> We are seeing the same
>
> It looks like an odd failure. Data coming through a few messages at a time every 10 min

It stopped entirely around 2320 last night and the only acknowledgement of a problem is this, which appears unobtrusively on the status page:

"2023-07-16 - A reduced/intermittent message rate is being experienced on the approved feed platform. The platform is currently being restarted”

Nothing on Twitter (despite the fact they’ve tweeted about outages to the platform on there before).

As soon as I have time to migrate over (paid work takes priority), I’m moving off approveddatafeeds on to a direct feed from Network Rail. CACI have had numerous chances over several years and again have proved themselves again incapable of operating the service properly.


Peter

[Peter Hicks]

And it's back up.  For those who might have missed it, from the status page:

"2023-07-16 - A full restart of the approved feeds platform has been carried out, normal service should have now resumed apologies for any inconvienience that this may have caused."

Peter

[QRail Comms]

It is indeed back up and running. It's a shame there was no reply to the ticket (though that will no doubt come on Monday with the same text as above.) Things appear normal for the last couple of hours at least so maybe the Monday peak won't be missed after all!

Lucky that OTT have the option of a direct NR connection, but what hope is there for other established users who don't have that luxury.

CACI have shown from the start little interest in the open data service, and it would appear very much that Network Rail & others have allowed that to happen. There is a slim hope that the rail data market place may help with some of these problems, but that really depends how high the bar for entry is set and what (if any) useful documentation is released.

[Phil Wieland]

I got a reply to my ticket at 12:15.

I'm now wondering how to switch back to approveddatafeeds without losing any messages.

[Peter Hicks (Poggs)]

> On 16 Jul 2023, at 14:47, QRail Comms <qrail...@gmail.com> wrote:
>
> It is indeed back up and running. It's a shame there was no reply to the ticket (though that will no doubt come on Monday with the same text as above.) Things appear normal for the last couple of hours at least so maybe the Monday peak won't be missed after all!
>
> Lucky that OTT have the option of a direct NR connection, but what hope is there for other established users who don't have that luxury.
> CACI have shown from the start little interest in the open data service, and it would appear very much that Network Rail & others have allowed that to happen. There is a slim hope that the rail data market place may help with some of these problems, but that really depends how high the bar for entry is set and what (if any) useful documentation is released.

I’ve considered a direct connection numerous times before, but it means I’ll lose the ability to stand up and shout when the Open Data platform goes down, which I’ve done consistently for everyone’s benefit, not just mine. That is a very important thing to me.

Connecting directly to Network Rail is non-trivial, not a quick process and requires technology that is unfamiliar to people who haven’t worked in an enterprise environment before. I’ve invested about £25k in software licenses to be able to do this robustly over the last six or seven years and aside from a couple of issues with AWS’ infrastructure and some full disks, downtime has been minimal. You can do it much cheaper than that, although you’re on your own (or in the realms of paid-for consultancy) if you want to make good use of it. Email me off-list and I can put you in touch with somebody at NR who can talk you through what’s needed.

RDM should be releasing the same feeds as NROD, ideally using the same protocols but with a much better level of availability and support. I will have a word with RDG and see if I can get them to publish something on timescales and progress.


Peter

[Tom Cairns]

I’ve had a very busy weekend so putting my hand up now.

 

There has been a relatively open forum that Peter invited others to on this list (must have been about a year ago) where CACI have attended with NR staff related with the open data channels.  There have been a number of questions in recent sessions where the responses have been concerning at best, and I have passed my comments onto NR accordingly.

 

Like Peter, I have been somewhat reluctant to shift across to the direct feeds to NR. Some of RTT does use those feeds for other purposes and I have a copy of it running on those feeds for industry use. This weekend was somewhat of a last straw and I shifted it over to our direct ingress after a few hours of outage. It is my intention to move it back overnight at a point once I have some feedback from NR and CACI – as having multiple high profile services affected is critical to get the attention on it.

 

The latest feedback I’ve had about NROD and RDM from NR is that they are taking a view of ‘wait and see’ with respect to real-time feeds. The default licences with RDM will currently require entities to agree and comply with the normal additional rules that NR do not currently have – I haven’t yet had feedback as to whether NR will follow these default terms but RDG told me verbally that “we won’t enforce it” to which my response remains “why do those terms exist at all then?”

 

Tom

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/63C0927C-A7C4-455D-987E-F08AB9CB7D88%40poggs.co.uk.

[Peter Hicks (Poggs)]

Here’s a reply to the ticket I logged on Saturday:

   "We were aware of a problem on the the approved system on Sunday morning and services were investigated and subequently restarted.  Normal services should have been resumed by approximately 11:50am."

It takes some nerve to write a reply like that.

Peter

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/PRAPR02MB7858960A974BD8F1AD1FA709E73AA%40PRAPR02MB7858.eurprd02.prod.outlook.com.

[James Uppington]

I can’t help thinking that there’s something severely lacking in their monitoring capability.

They only seemingly become aware of issues when people complain, as has always been the case.

Also, if the root causes of the instability aren't solvable, surely by now there should be enough of a catalogue of modes of failure vs. remedy that recovery should be automated, reducing the TTR for most failures.

I’ve been called in the early hours of the morning enough times to build automated repair/recovery into everything I write as a priority nowadays.

From James Uppington

upp...@me.com

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/A064C37A-E52C-4831-8915-8ED5CA07B024%40poggs.co.uk.

[QRail Comms]

To echo Peter's reply - we got exactly the same stock response to the ticket this morning... Not helpful & no clue as to what happened or what's been done to make sure it wont happen again. Gives absolutely zero confidence in there service provision.

Re NR direct; £25k is prob a bit much for people without significant commercial relationships to absorb. Camel (for example will talk to MQ rather than having to pay for commercial MQ licenses) but no idea if that's allowed/supported. Fingers crossed for RDM to be able to deliver something.

Phil Re switching back between the feeds - I don't think that's possible without loosing some messages, but you could do it very late at night so any loss would be minor (from the TD side) and if you time it right you shouldn't drop much TRUST stuff either.

Sadly we only use/monitor the approved feeds, the account on the public feed lapsed so can't test anything right now.

And to echo what James says; They have no clue about monitoring! Look at how long it took to get a basic graphana page up and running and they still can't deal with an issue before everyone starts to complain. It's not hard to have an email alert setup. But remember as far as CACI as concerned this is free service so what's the point in that waking someone up at 3am and having to pay them money to work on something that won't directly affect there bottom line. I suspect if one of there commercial partners had an issue there would be a much prompter response... Again we blame Network Rail and others for allowing this sad state of affairs.

[Tom Cairns]

I think that this a useful opportunity to ask for an open call about the issues that everyone current feel they are facing with the platform.

 

NR do read this mailing list and asked me last week to provide a list from my point of view and if possible what I could summarise from the group’s recent discussions. So – please go for it and I will pass a list over to NR in the next few days.

 

Tom

.  

 

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/6ccaafea-ea90-48df-a787-5093380a2d35n%40googlegroups.com.

[Phil Wieland]

 

Phil Re switching back between the feeds - I don't think that's possible without loosing some messages, but you could do it very late at night so any loss would be minor (from the TD side) and if you time it right you shouldn't drop much TRUST stuff either.

That's what I thought.  Next time I'm on line at 3am ...

Thanks,

Phil

[QRail Comms]

Absolutely agree, but we need a few more people than just those who post here. Frankly the entire community engagement needs looking at and has done for some time now. There are people out there who would like to feed back but don't want to do it via here or twitter (And its not like CACI listen, and NR have become rather 'flaky' since a couple of specific people left a while ago.)

For example its a shame the recent RDG survey was not sent directly to each registered user of the open data platform (the details can be obtained) rather than just a post to this list. In our last catch up with the RDG it was shocking (to us and them) at the lack of replies! but was also good to see a lot of the stuff that did come up is the same across the board. Documentation, support, and a service which basically works and has faults responded to in a timely fashion.

It would be nice if someone from NR & the RDG made the occasional post or replied to queries here. Its one thing to just read through, its quite another to actually get involved and interactive with the community (which would be a good thing right now)

Another good example question is; how does an established user get on to the approved data feeds platform rather than the public version? What ever happened to a sandbox for new users? I'm sure there are many more.

It's prob worth starting a new thread with a call for info!