Notes on RDM Darwin PushPort

[Chris Bagnall]

Hello all,

With the pending closure of the NRDP PushPort feed, I have been working on updating our interface that accepts data from PushPort to use Kafka instead of ActiveMQ.  Here are some notes that may be of use to others.

First some notes on Kafka:

• Kafka topics are split into partitions.  Each partition acts as a queue, but Kafka clients switch between the partitions in a manner that does not allow delivery order to be guaranteed.
• The RDM Kafka topic has two partitions.  
  • In theory, this limits us to a single client as you normally want to have at least two partitions per client to avoid a client being idle. 
  • In practice, there is so much data coming in on both partitions that this will not be the case.
  • You will still want to just have a single client if you want to sort out message ordering (see below).

There is a known issue where events for a single train can be found in both partitions.  This has been reported on:

• https://groups.google.com/g/openraildata-talk/c/oth6M9vnsKU
• https://raildata.org.uk/community/category/topic/postDetail/94

As a result of this issue, I have been looking at implementing a buffer that uses the PushPort sequence number to enforce the correct ordering of events and detect any missing messages.

Observations based on testing the RDM PushPort Kafka feed:

• The Kafka feed offers a much longer history of events when compared to the ActiveMQ feed
  • You can access this history on startup/restart by setting the auto.offset.reset property to earliest
  • This will start retrieving events from between 60 and 150 minutes in the past (as opposed to 5 minutes - IIRC - for the ActiveMQ feed)
• The flush time for the Kafka feed appears to vary between partitions.
  • In my last test, I had a 150 minute history on partition 1 but only a 120 minute history on partition 0
  • This guarantees that you will have missing events until you have processed the first 30 minutes of partition 1's data.
• Sometimes the messages come in out of sequence even within a partition.  Here is an example from when my client was catching up:
  • Latency here is the time the event was received by the client minus the timestamp contained in the JSON envelope.
• Even when not processing the history, there can be some latency.
  • Usually this is sub-second, with occasional spikes up to 5 seconds.
  • I have seen the latency get as high as 58 seconds.
  • I think that the higher the latency, the more likely you are to get messages out of sequence in the same partition.

Hopefully these observations will be of interest/use.

Cheers,

Chris

[Chris Bagnall]

Apologies for the image not posting.  Hopefully this will work to show the out of sequence messages on the same partition.

[Peter Hicks]

Hi Chris

I’d actually suggest you don’t try to get around the incorrect partitioning on RDM in an effort to encourage the issue to be fixed upstream.  However balanced against that is your need to be ahead of the game and not tripping up when NRDP is switched off.

I don’t have visibility on where fix is for this issue, and I hoped it would have been a really straightforward fix.  Seemingly not a quick one.

Peter

Sent from Proton Mail for iOS.

-------- Original Message --------

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/049f3f38-f53a-450c-ba09-8da6f8454aeen%40googlegroups.com.

[Jez Smith]

Peter, Chris,

The fix (bringing the feed into a single partition) is currently in development and has been slightly delayed as we consider load / capacity.  We are planning to put the change into production at 2am on Friday 20th March and will monitor performance.   

We are also working with our supplier on developing 'snapshot' functionality.  We are currently defining the requirement and design, so it is further off, but we have allocated funding for it, so it will be going ahead.

Cheers

Jez

[Marks Polakovs]

Just saw this update come through:

> To ensure consumers receive a correctly ordered stream of messages, we’re moving the Darwin Real Time Train Information pub/sub feed to a single partition.The change is scheduled for 2 AM (UK time) on Friday 20 March. The feed may be unavailable for up to one hour during deployment, after which we will monitor performance.No action is required from subscribers, and we do not anticipate any impact to your systems once the feed resumes.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/734d9b69-ba35-44a4-83a3-43abef6e54cdn%40googlegroups.com.

[Chris Bagnall]

Hi Jez,

Thanks for the note.  As with Marks, I got the update on Friday.  Dropping down to a single partition is the easiest option, if somewhat of a brute force solution.  

I will check on the 20th to see if this also solves the messages coming in out of order on a single partition issue is also solved by this change.

I look forward to seeing the snapshot functionality.  That will be useful on the odd occasions when we miss a message.

Cheers,

Chris