PushPort questions

[Clint C]

HI guys

sorry if this might of been asked before.

i would like to know how many XML messages to expect per minute at worse case scenario.

also does anyone know how long after a service reaches its destination that a deactivate message comes?


thanks for your help

Clint.

[Peter Hicks (Poggs)]

Hi Clint

On 13 May 2018, at 00:21, Clint C <cl...@net-key.co.uk> wrote:

i would like to know how many XML messages to expect per minute at worse case scenario.

This is the throughput graph on my end - looks like ~30 message/sec

also does anyone know how long after a service reaches its destination that a deactivate message comes?

To be honest, I’m not sure - it’s probably one or two hours after arrival at destination, but you’d be best to check yourself on a day’s worth of data.

Peter

OpenTrainTimes Ltd. registered in England and Wales, company no. 09504022.

Registered office: 13a Davenant Road, Upper Holloway, London N19 3NW

[Jon Cox]

Hi Clint,

It was a couple of years ago that I was playing around with the Push Port, but when I measured it I too got an average of about 30 messages a second.

As you might be interested I've attached some logs I made of the average number of messages per second over the previous minute. It was just a quick hack to get a rough idea of the number of messages coming through and doesn't cover the worst case scenario, but you can see that during normal operation it does reach 50+ messages minute on occasion. (At least it did 2 years ago.)

Jon

[Clint C]

thank you guys, great help

jon - logs very useful, may be a good idea to do something like this myself

Peter - i love that graph, i want one! but i cant figure out how you reconcile to 30 m/s?

[Clint C]

Hi guys

i have my database up and running and pulling data from the pushport. So far so good.

because i am using php it is not possible to have a process that runs indefinitely just waiting for data, so i have created a process that runs on a schedule every 120 secs and the process stops after 110 secs. (to be relaunched 10secs later).

the concern i have is how can tell if i miss data? If there is a connection issue, how how long a gap between connecting before the server stops queuing for me? and at what point i need to clear the database and go and get new FTP data and start over?

thanks in advance.

Clint

[Peter Hicks (Poggs)]

Hi Clint

The TTL on messages is set to either 10 or 15 minutes, I can’t quite remember which. Disconnecting every 110 seconds and reconnecting again will work fine, but bear in mind the following:

* Your consumer will need to reconnect promptly, within a couple of minutes, otherwise you risk not being able to process the backlog of messages plus any new ones queued
* If your consumer fails to reconnect and messages expire on your queue, you’ll need to download the logs from the FTP server for the duration you missed

There is probably a message header set with the time messages were queued, and/or the time at which a particular message expires. Have a look at these and if your clock is set correctly on your local machine, you may be able to see when you’re receiving messages with a dangerously low time-to-live.

A better option for you might be to use an ActiveMQ bridge - where ActiveMQ running on your side is permanently connected to the NRDP server and re-queues messages locally on to either a queue or a topic. This way, you can buffer more than 10/15 minutes worth of messages locally. It also has the advantage that you can split out the incoming messages in to multiple queues, or a single topic with multiple subscribers, and use the same feed for your live, pre-prod and development environments.

Kind regards,


Peter
--

[Rail Delivery Group]

The queue length is 10 mins. On the connect/re-connect process you mention, you may want to look at using a different language to php as I believe multiple connections/disconnections has caused memory issues with the NRDP service in the past, causing it to fail for all consumers and ultimately resulted in IPs using that method being blocked.

I might be wrong and it might be unrelated and your approach is fine (maybe David or someone else from CACI will see this and comment) but just making you aware this is a possibility...

[Peter Hicks (Poggs)]

> On 25 May 2018, at 10:37, 'Rail Delivery Group' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:
>
> The queue length is 10 mins. On the connect/re-connect process you mention, you may want to look at using a different language to php as I believe multiple connections/disconnections has caused memory issues with the NRDP service in the past, causing it to fail for all consumers and ultimately resulted in IPs using that method being blocked.
>

> I might be wrong and it might be unrelated and your approach is fine (maybe David or someone else from CACI will see this and comment) but just making you aware this is a possibility…

Disconnecting and reconnecting is fine at intervals of, say, two minutes - but the problems occur when a user doesn’t disconnect, and then connects again. This causes an increasingly large volume of connections to a queue from one IP address, and that’s what causes the issue.

I had provided details a method of limiting the number of simultaneous TCP connections from a single IP address to ’n’, but I don’t think this ever got implemented.