Push Port running behind?

[Tom Thorpe]

Hi,

Is anyone else noticing the Push Port (the stomp queue that you access with the details from datafeeds.nationalrail.co.uk) running a few minutes behind at the moment? I noticed it yesterday evening and also this morning, it seems to sometimes run up to 5 minutes behind.

I thought maybe it was my client not picking up the messages in time, but then I checked the ftp files and downloading the pPortData.log file. That file shows a modified time of now, but the latest entry is a few minutes behind too - which to me indicates its being written with data that's a few minutes behind?

It does seem like it's drifting and sometimes catching up a bit, but then falling back behind again...

Thanks,

Tom

[Phil Wieland]

Unfortunately I don't log latency so I can't comment about earlier today, but I can say that as at 10:12 today, my TRUST and TD feeds are bang up to date.

Which topics are you having trouble with?

Phil

[Peter Hicks]

Hi Tom

Can you report this to support and let us all know the outcome?

Cheers,

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[petermount]

I don't log latency either but the last 3 hours have seen some odd message rates, looks like 2 drop offs and then it catches up which might explain what you are seeing.

From my Monitoring, it should be fairly smooth on the right but there's a peak. Looks like a drop off about 3 hours ago then catches up & now dropped again.

[Tom Thorpe]

Yep, that definitely looks about right. Good to see, I was concerned about whether it was just my client or whether others were experiencing the same.

I did contact support and I've just received this reply:

-----
There was a planned outage from Darwin (the system which we connect to for the feeds). which required them to undertake a failover test of the network due to a recent deployment on their side.

This might have been causing the delay and some connections problems as well.

Sorry for the inconvenience.

-----

As of right now it's keeping up to date, so if it stays that way hopefully it's resolved :-)

Thanks everyone!

Tom

[petermount]

I knew of the fail over tests on Wednesday night but nothing about them since then.

That said there were a few "incidents" last night where I saw the push port feed drop off so much my client restarted itself, so it could have been that.

For everyone else, I've been adding some better monitoring which might be useful if you are wondering if there's an issue affecting anyone else - i.e. it's what I'm seeing.

If you go to http://status.area51.onl/ take a look at the "Data Feeds" box & hover over the 15m value for the feed. You'll then see a graph of the last 24 hours (from that moment) of what my system has seen.

It's pretty basic but it might be useful.

Peter

[Tom Thorpe]

That looks like it'll be very useful - thanks!

I'll get in touch with support if I notice it fall behind again and report back here.

Cheers,

Tom

[Peter Hicks]

Hi Tom

On 30 Oct 2015, at 10:55, Tom Thorpe <sp...@tomthorpe.co.uk> wrote:

I did contact support and I've just received this reply:

-----
There was a planned outage from Darwin (the system which we connect to for the feeds). which required them to undertake a failover test of the network due to a recent deployment on their side.

This might have been causing the delay and some connections problems as well.

Sorry for the inconvenience.

-----

As of right now it's keeping up to date, so if it stays that way hopefully it's resolved :-)

Quoting from the email that National Rail Enquiries sent out:

"During the evening on Wednesday 28th October @ 22:00 one of our 3rd party Suppliers is required to undertake a failover test of the network, for the majority this will be a transparent test however there are some who will experience a brief disconnection from Darwin both on failover and on failback.  Also during this time there is a possibility that Trust and TD feeds into Darwin may cease"

If there were latency issues caused by this failover test, I’d expect them to have been apparent from Wednesday night and all through Thursday - was this what you were seeing?

Peter

[Tom Thorpe]

Hi Peter,

Almost. I saw it happening all yesterday (as would be expected from that email) but also this morning, which is when I got concerned.

As of right now though it does seem to be up to date again, so perhaps this morning was just a bit of a fallout from the testing? 

Tom

[petermount]

I've seen a few disconnects - other than the usual  early hours when Darwin's restarted:

28th:

• 13:03
• 16:23
• 17:02

29th:

• 19:07 - 19:14
• 19:42 - 19:58
• 20:25
• 20:45

Where there's 2 time above thats the start & end of the 'outage' - i.e. it tries to reconnect & nothing comes back.

In this instance the watchdog restarts the connection after 3 minutes of no messages being received.

That said I doubt it's due to that outage as I do see this happening a bit for the last week.

Peter

[Peter Hicks]

Hi Tom

On 30 Oct 2015, at 13:20, Tom Thorpe <em...@tomthorpe.co.uk> wrote:

> Almost. I saw it happening all yesterday (as would be expected from that email) but also this morning, which is when I got concerned.
>
> As of right now though it does seem to be up to date again, so perhaps this morning was just a bit of a fallout from the testing?

I’m not convinced it’s directly related to the testing - the whole platform appears to be fragile.

Has anyone else seen quality issues with the Darwin feeds?


Peter

[Tom Thorpe]

It seems to be running slow again unfortunately at the moment (in the attached screenshot the latest FTP modified time is 18:10, but the latest item in that file is 18:04 which you can just about see highlighted in the bottom right - I dont actually use the FTP feeds I use the stomp client, but im using this to make sure it's not just my client picking up the queue too slow).

I'll send this to my original support ticket and see what they say.

Tom

[Tom Thorpe]

Looks like it's now almost 15 mins behind. Hopefully they'll get a chance to look at it on Monday!

To confirm, the push port is supposed to be pretty much realtime, right? I'm hoping this won't be deemed as within the expected behaviour.

[Peter Hicks]

It needs to be real time otherwise you've got no chance of displaying information consistent with Darwin...

P

[Tom Thorpe]

Is anyone else noticing this is running slow again this evening?

I'll open a ticket now.

Tom

On Friday, October 30, 2015 at 9:40:21 AM UTC, Tom Thorpe wrote:

[Peter Hicks]

Hi Tom

"Slow" may be an understatement.  It's taking me 35 seconds to log in to the UI, and I can't even FTP in.

Please, log a ticket :)

Peter

--

[Tom Thorpe]

Thanks Peter! I've opened a ticket and I'll post back here when I get any info

Tom

[Tom Thorpe]

It seems to be almost an hour behind for me now at the moment. My ticket didnt get a reply (although I did get a ticket number).

Does anyone know how I might be able to get more help please? And is anyone else getting delayed data on the push port stomp client?

Cheers,

Tom

On Friday, October 30, 2015 at 9:40:21 AM UTC, Tom Thorpe wrote:

[Peter Hicks]

Hi Tom

Try logging another ticket - its tricky, as an outsider, to get somebody to pay attention to a problem.

One question - are the push port logs showing up as delayed too?

Peter

--

[Tom Thorpe]

Hi Peter,

Thanks - I'll try that.

Do you mean the logs in the FTP? Yes they seem to be coming through delayed in the pPortData.log file too. Also if I disconnect the stomp client and wait 5 mins or more (as I understood it, data is only kept in the queue for 5 mins?), it still picks up from the delayed time, currently almost an hour ago. So unless I'm misunderstanding, that seems to indicate something is wrong

Hopefully another ticket might work. Thanks for your help as always!

Tom

[Peter Hicks]

Hi Tom

I've just replicated the problem on my end - messages from 12.50pm coming through at the moment.

Good luck!

Peter

[Tom Thorpe]

Thanks for replicating it - it's good to know it's not something on my end.

[Tom Thorpe]

Hey,

There seems to have been big issues again this weekend, with it being over over 2 hours delayed at one point yesterday.

I've started logging the delays on my feed processor. As messages come in it compares the timestamp on the message with the current time, then roughly every 2 minutes records the latest difference to the log. I've attached the output of the log from this weekend, it's interesting to see the delay gradually increasing throughout each day. The worst of it was getting to 132 mins delayed at 19:16 on 11th December before then receiving no messages at all until 21:08 (hence the gap in the logs) when it then resumed messages up to date, but gradually started getting delayed again until about half 10 at night.

I've sent the file to the support email to see if they can hopefully find out what is going on with the feed and why this keeps happening. Fingers crossed! :-)

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-talk+unsubscribe@googlegroups.com.
To post to this group, send email to openraildata-talk@googlegroups.com.

[petermount]

it seems to be doing the same thing again today.

The last message I've received was at 09:46 and that had a ts of 2015-12-14T09:00:33.406758Z

Peter

To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.

[Tom Thorpe]

I noticed the same too. I've sent another ticket to them this morning but haven't had replies to the ticket from the weekend or this morning yet (other than the automated responses giving me a ticket number).

Maybe if anyone else could also open a ticket with them please it might help them spot it?

Cheers,

Tom