Access to Train Movements product on RDM

[Александр Цыплаков]

Good day all!

I have just run into a strange problem: 
- I've subscribed to the NWR Train Movement product and setup the listner code to record messages. Last time it worked and saved messages was this morning around 6 am
- This evening same code just straight up fails to connect to the server, returning the following error

//Connection setup timed out in state SSL_HANDSHAKE (after 30195ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)

My question: is it possible that I has been "switched off" from the product  by its provider for some reason? 
I am lost in regards to what else could be causing this: my code is exactly the same as this morning, same computer, and the Network Rail outage dashboard shows no current outages of TRUST (as far as I could read it).

Best regards,
Alexander


 

[Peter Hicks]

Hi Alexander

If your connection is timing out after 30 seconds, it sounds like you're unable to reach the Kafka service.  If it were a problem with an SSL certificate, or invalid credentials, I'd expect it to fail rather more quickly.

One possible issue is that your IP address is being blocked from accessing Kafka.  A quick thing to try is connecting from a different host - preferably in a different geographical location if you can.  I have a feeling you may have ended up having your IP address incorrectly geolocated to a territory not permitted to access the service - this happened to somebody else, whose IP address ended up being geolocated to Russia, but was clearly Germany.

Peter

[Александр Цыплаков]

Thank you, Peter. I will try to do that.
I've also written an email to Open Data Team at NR - I know they are on Christmas break, but maybe it would help.

I have a small theory on the cause: when I was trying to understand the structure of data for the messages, I was running the script from my corp. laptop - which is in London. 
However, later I tried running the script from a corporate virtual machine, that is on our US server
(this was the sollution our tech. people suggested for running a listener 24/7)
That try didn't work - connection timed out in exactly the same way.

And now connection times out what I am trying to run it from my corp laptop again.
I can't imagine that US would be one of the "restricted countries" - but maybe my account was flagged for using IP addresses from 2 different countries (and continents, for that matter).  

Regards,
Alex

понедельник, 29 декабря 2025 г. в 10:23:03 UTC, Peter Hicks:

[Peter Hicks]

On Monday, 29 December 2025 at 10:48, Александр Цыплаков <deli...@gmail.com> wrote:

I've also written an email to Open Data Team at NR - I know they are on Christmas break, but maybe it would help.

Likely the NR team would forward you on to the RDM team at RDG - who are operating a limited service due to annual leave at the moment.

I have a small theory on the cause: when I was trying to understand the structure of data for the messages, I was running the script from my corp. laptop - which is in London.
However, later I tried running the script from a corporate virtual machine, that is on our US server
(this was the sollution our tech. people suggested for running a listener 24/7)
That try didn't work - connection timed out in exactly the same way.

And now connection times out what I am trying to run it from my corp laptop again.
I can't imagine that US would be one of the "restricted countries" - but maybe my account was flagged for using IP addresses from 2 different countries (and continents, for that matter).

If your account was flagged as suspicious for connecting from two different addresses from two different continents, I'd expect something to be done to your account, rather than both IP addresses to be blocked.  To take a real-life example - yesterday, one of my Microsoft 365 accounts was blocked for suspicious activity when I logged on from a train.  Had Microsoft blocked all access from that IP address, they would have impacted many more users than just me.  Instead, that account was prevented from logging in until the problem was resolved, so I knew it was an account issue.

Trying a connection with your client from an IP address you've not used before - if that's possible with your setup (e.g. your laptop may have an 'always on' VPN such as ZScaler which will always tunnel traffic save for any captive portals) - will rule out an issue with your software itself.  Unfortunately, corporate networking and security restrictions can often get in the way of software development and you might even find that running the software from your corporate laptop, even not going through corporate infrastructure, might fail due to a locally-enforced firewall.

Peter

[Александр Цыплаков]

Peter, thank you for your recommendation - it worked!

I've moved my script to my personal laptop - that is 5 meters away from my corporate laptop - and now it runs smoothly.
It must have been something in the ZScaller settings, that was/still is tripping up the Kafka server. 

Regards,
Alex

понедельник, 29 декабря 2025 г. в 11:48:49 UTC, Peter Hicks:

[Mike F (Hastings Line)]

IP authentication can be hit and miss with ZScaler in my experience, due to recycling and reassignment of IP addresses

M

[Александр Цыплаков]

If I may, I want to ask a quick question which does not  really  merit a separate thread: 
- I have about 20 hours worth of messages from Train Movement (TRUST) - and it sums up to about 27 thousand messages

It feels too low for reality.
Specifically because almost half of train_id's (the 10-digit TRUST train ids) only having 1 message per day as a result.

How many Train Movement messages should I be expecting per a full day - at least in orders of magnitude ?
(10's of thousands? 100's thousands?) 
Wiki states Train Movement produces 600 messages per minute at peak - which would be 36 000 per one hour alone...

Regards,

Alex  

вторник, 30 декабря 2025 г. в 15:17:32 UTC, Mike F (Hastings Line):

[Peter Hicks]

Hi Alex

On Tuesday, 30 December 2025 at 16:59, Александр Цыплаков <deli...@gmail.com> wrote:

If I may, I want to ask a quick question which does not really merit a separate thread:
- I have about 20 hours worth of messages from Train Movement (TRUST) - and it sums up to about 27 thousand messages

It feels too low for reality.
Specifically because almost half of train_id's (the 10-digit TRUST train ids) only having 1 message per day as a result.

How many Train Movement messages should I be expecting per a full day - at least in orders of magnitude ?
(10's of thousands? 100's thousands?)

Wiki states Train Movement produces 600 messages per minute at peak - which would be 36 000 per one hour alone...

Taking data from 19th December, I received 813,485 messages on the TRUST feed.  Even on Christmas Day (25th December), I had 61,648 messages.  These don't differentiate between train movement messages and activation, cancellation, reinstatement, change-of-origin, change-of-identity and change-of-location messages - but movement messages are the most prevalent on a typical day.

Have you done a cursory check on message latency, taking the timestamp in the XML messages and comparing it to the time you received it?  When you're pulling messages live, it should be in single or at worst double digits numbers of seconds.  If it's more than that, it sounds like an issue somewhere.

Peter

[Александр Цыплаков]

Yes, I think I found the problem. A very dumb mistake in code on my part.
When messages come, they come in batches - and I am pretty sure my code was saving just the first message in each batch, instead of entire batch. 

I've corrected for it, now I'll collect the next full hour (from 17:55 to 18:55) and compare result to yesterday's same hour. 
If new tally jumps by x20-x30 compared to old tally - then I'm likely on the right path.

Regards,

Alex

вторник, 30 декабря 2025 г. в 17:04:31 UTC, Peter Hicks: