TD and TRUST feed instability

[Peter Hicks]

All,

Does anyone else have users of their services complaining about missing train movement messages/cancellations, or missing train describer data?

It looks like the TD and TRUST feeds have been problematic all day:

Compare this to a normal day:

I've reported this to the support people, but it'd be really good if anyone else here affected could reply to this thread of the impact it has on your particular services/products.

Peter

[Peter Mount]

Here's mine for today. Careering been fine except for an odd blip this morning, probably local to me, but the nr feeds have been a bit erratic.

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-talk+unsubscribe@googlegroups.com.
To post to this group, send email to openraildata-talk@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adrian Bradshaw]

Peter

 

Absolutely, yes, it's been very unstable, with missed messages and latency up and down like a yo-yo. Our diagrams have been littered with orphaned headcodes and on occasion minutes delayed compared to our live cameras.

 

I suppose it says something about our expectations that nobody even bothers to report it here. To be fair though, it's actually not been so bad of late...

 

Adrian

---

Adrian Bradshaw – Administrator (volunteer)

Railcam UK Limited

@: adr...@railcam.uk

Railcam.uk – Bringing The Lineside To Your Armchair

Railcam UK Limited | A company limited by guarantee | registration number 10167844

3rd Floor | 207 Regent Street | London W1B 3HH

Railcam UK Ltd is operated on a not-for-profit basis entirely by volunteers

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.

[David Higginbottom]

All,

Apologies for the latency and related issues today.
Everything should be resolved now.

Kind regards
Dave

---

From: openrail...@googlegroups.com <openrail...@googlegroups.com> on behalf of Adrian Bradshaw <adr...@railcam.uk>
Sent: Thursday, July 6, 2017 9:22:28 PM
To: openrail...@googlegroups.com
Subject: [SOCIAL NETWORK] Re: [openraildata-talk] TD and TRUST feed instability

 

This electronic message contains information from CACI International Inc or
subsidiary companies, which may be confidential, proprietary,
privileged or otherwise protected from disclosure.  The information is
intended to be used solely by the recipient(s) named above.  If you are not
an intended recipient, be aware that any review, disclosure, copying,
distribution or use of this transmission or its contents is prohibited.  If
you have received this transmission in error, please notify us immediately
at postm...@caci.co.uk
Viruses: Although we have taken steps to ensure that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

CACI Limited. Registered in England & Wales. Registration No. 1649776. CACI House, Avonmore Road, London, W14 8TS

[Mark Rawlinson]

Seems to be happening yet again today - TD feed suffering from ever increasing latency 

(over 7 minutes at the moment, according to my calculations)

Can anyone explain why this latency issue occurs - inadequate buffering and/or processing

power to 'catch up' after feed interruptions, perhaps, or is the raw data coming through from

the Network Rail TD system with latency to begin with??

MARK 

[petermount]

Yeah, looks like from about 1pm BST although it does look like there was another blip around 10am.

[Mark Rawlinson]

Seems to be kicking off yet again - latency has been 'choppy' since around 16.55

MARK 

[petermount]

I'm seeing the same, from about 16:25 onwards

[Peter Hicks]

Same here, about 1635.

Peter

--

[sweary]

Yep.... the Openwire seems to be getting later and later (currently 500 seconds) with all messages arriving eventually but the STOMP has missing messages, presumably as over 300 seconds are dumped as you'd expect.

Sent from my Samsung Galaxy smartphone.

[petermount]

Just checked the TD latency, right now it's over 10 minutes

$ curl -s https://api.area51.onl/rail/2/td/latency/current | jq '.'

{

  "lastUpate": "2017-07-13T16:55:27.488Z",

  "min": 605582,

  "max": 621547,

  "value": 618488

[Mark Rawlinson]

Seems to have been getting steadily worse - only around 15 seconds at first,

but quite quickly increased to over five minutes, which is obviously completely

unacceptable for a supposedly 'real time' feed which a lot of websites and apps

rely on these days!!

Hopefully someone from CACI will perform their usual 'turn it off and on again'' fix shortly,

but would it not be possible for them to set up a script which can detect when the feed

processing has become bogged down and automatically restart it without having to rely

on manual intervention every time?

Even to a layman like me that seems like a blatantly obvious solution (or at the very least

a sensible first step which would significantly reduce the impact of those TD feed issues

to end users), so it seems bizarre that have not considered it yet??

MARK

[petermount]

I'm wondering if someone has flipped a switch as this is more like it.

$ curl -s https://api.area51.onl/rail/2/td/latency/current | jq '.'

{

  "lastUpate": "2017-07-13T17:12:59.781Z",

  "min": -403,

  "max": 2283,

  "value": 781

[Adrian Bradshaw (RAILCAM UK)]

It does sound like a sensible 'bodge', if nobody feels inclined or able to look into the root cause,

Adrian

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Phil Wieland]

I think it would be preferable to look at actually fixing the fault.  Perhaps they could start by considering why a four year old version of ActiveMQ is being used.

Phil