No longer receiving data from RDM

[John Avison]

I have noticed that I no longer receive any messages from the feed shown below.
Even more curious, is when using the 'Preview' button on RDM, it spins for about a minute, and then shows 'No Data', whereas previously, it used to show an example of a movement message.

Has something changed, or has my access been revoked or something, or is this just an ongoing issue?

For what it's worth, it appears that I have not received any messages since 9th February - just over 10 days..

[Matthew Burdett]

Hello John

Just checked this myself and I am also seeing the same problem.

Having checked train Describer as well, the problem is also on that channel too and most likely others.

Maybe worth raising a ticket, clearly there isn't yet a system in place that monitors and alerts for issues.

Kind regards 

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/566f5939-59c4-4ba8-bd7d-0aadc77584e4n%40googlegroups.com.

[John Avison]

Ticked raised - will update with any response

[Jon Wood]

Sounds like it could do with some sort of heartbeat monitor, that really isn't ideal.

FWIW the direct feed straight from the Network Rail site has had no significant issues that I know about, so it may be worth moving over if this persists. I believe the data is in the same format.

[John Avison]

I was originally using the direct feed, but suffered issues where I would essentially get timed out, and could no longer receive any messages, and then need to stop my client from connecting, and wait some time before reattempting (around an hour).
Moving over the the RDM feed stopped this issue, but this is the first time I've noticed it down. Having been away from work for a short while, I hadn't noticed this issue occur.

[John Avison]

Have not received a response, but, coincidentally  I have received an email shortly after saying that there will be planned maintenance this weekend which will require credentials updating, and may cause the exact issue I've described above.

It may be that nobody was aware of the issue, and once highlighted to them, they've decided to carry out this maintenance?

Who knows...

[Peter Hicks (Poggs)]

On 20 Feb 2025, at 15:53, John Avison <in...@shotbyaj.com> wrote:

Have not received a response, but, coincidentally  I have received an email shortly after saying that there will be planned maintenance this weekend which will require credentials updating, and may cause the exact issue I've described above.

It may be that nobody was aware of the issue, and once highlighted to them, they've decided to carry out this maintenance?

That seems unlikely - I would hope this would have gone through Change Management which takes several days at least in many of the smaller companies I’ve worked in.

The NR Train Movement feeds are, I believe, supplied by CACI.  I am not confident they will have any monitoring in place, so even if the problem lies elsewhere (e.g. having reached capacity on the hosted Kafka instance), I think they’d be unaware there was an issue until a user reported it.

I think I may have to resurrect my NR Datafeeds Monitor website to give a health indicator for whether the RDM-supplied feeds are working.

Peter

[Jez Smith]

OK, so let me give it a go....and Matthew is right, this does relate to the email recently sent out by RDM (who, for clarity, I work for!).  A little while ago, we worked with NR (and their delivery partner CACI) to put the NROD datasets onto RDM.  What we hadn't appreciated was that there appeared to be some pent-up demand, the result of which was that we ran out of access keys to assign (which we became aware of last week).  We are busy fixing that, so on Saturday we will increase our available keys by an order of magnitude.  So we were aware that this meant that we could not provide keys to any new subscribers until after the upgrade, but all existing subscribers should not be affected.....however....

What we didn't realise was that in the mean time a manually generated key assigned to CACI expired (who provide the train movement data) and there wasn't an available key.  So we are currently trying to work out how to create an additional key and get that dataset back on line.  A solution is still tbc.  It will definitely get resolved on Saturday by the upgrade, but I am hoping that we can resolve it quicker than that.

My apologies to those experiencing difficulties with this dataset and to the general inconvenience of new keys on Saturday.  

John raises good points about monitoring, and we will look at what we can do once we have resolved the immediate difficulties.

Cheers

Jez

[Christoper Stafford]

Were you sending heartbeat messages? I noticed that when they upgraded ActiveMQ that the server started needing a message from the client every minute or so to keep the connection open. If you were reconnecting every minute then it's quite possible that you would get rate-limited off the server.

In principle I think the RDM feed may well be the more reliable and durable feed to be on, but thought I'd offer a possible reason why the other feed stopped working.

On Thursday, 20 February 2025 at 15:12:33 UTC John Avison wrote:

[John Avison]

OK, so shortly after the last email received, I received another one directly to me which said that they had looked into my query, and had confirmed that there was an issue, and quickly came up with a solution to roll out the fix this weekend.

They basically thanked me for pointing it out, and said I should have also received the email which was sent to everyone, as they had confirmed I was on the mailing list.

Primarily a software developer, I work alongside NWR via their delivery partners, and I am meant to be showcasing a new system next quarter which I have been working on, which utilises the TRUST data.

What has been helpful, is that I have direct access to TOPS/TRUST and have been able to query TRUST to ensure everything matches up with the new system.

I did have a monitoring system in place, but unfortunately, it just wrote to a log file.

I'll make changes to actively monitor and alert via email. to my NWR address.

All in all, everything looks to be getting resolved, and it's business as usual from this weekend.

[Matthew Burdett]

Guessing still waiting for caci to update their keys as this still isn't working for me, unless it's just me?

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/bb3dae13-dbd2-463e-bd8e-d102f3df6394n%40googlegroups.com.

[DwayneDibley]

I have just given it a try seeing no data. Have filed a support ticket and waiting a response.

Jack

[Jez Smith]

Quick update on where we are with this issue; the infrastructure upgrades were successful on Saturday, so we now have sufficient keys for the foreseeable future.  The swap-out of the keys for the CACI team as the data source publisher wasn't as straightforward as hoped resulting in a few technical calls, some updates to their code and a bit of testing.  All now seems to be resolved, and they are hoping that the data products will be available again by the end of the day.

[Jez Smith]

The data should be available now, so if it isn't, please let us know by raising a ticket on RDM.

Jez