Duplicate and missing Pushport messages

[Paul Clark]

We are consuming TS and SC messages from pushport but seeing a lot of duplicate messages - one train had 3146 duplicate messages when there should have been 148 unique messages. I've attached an example of a train with just a few duplicates in MyReceivedEvents.txt (e.g. lines 1-2, line 9-11)

We've also compared our messages to the archive here:  https://archive.area51.dev/archive/ukrail/darwin/

The archive also has duplication but to a much lower level than us but there are also missing messages on both sides. e.g. in one example, the archive had 25 messages and we received 264 unique messages – the archive was missing 246 messages and we were missing 7.

We're using a client based on this: https://github.com/openraildata/stomp-client-dotnet

(note: we're using the current version of the repo - OpenWire, not Stomp).

Has anyone else seen duplication or done a comparison to the archive?

[petermount]

One thing about the messages in the archive, the messages are filed under the time they arrive at the associated logger and not the time the message was received/pertains to, so if the messages were delayed then the could be recorded under a following minute log file, which might be why you didn't find them if you were not looking at later times.

Possible causes for are:

1. Messages received out of sync - on my end I check each message to ensure they are in sequence. If they are not, then I re-sync from the static state before processing following messages. This usually catches missing data but could cause duplicates in the archive.
2. Messages delayed due to upstream latency so they are archived in later files.
3. Messages not received from upstream due to an issue at NRE.
4. Messages being delayed when received by the logger due to network issues between the core and the logger.

Notes:

1. For historical reasons, the core (which receives from NRE and does the processing) is in London whilst the loggers are in Amsterdam, so network issues in-between the two can cause delays.
2. The loggers just record the messages as-is as they don't know what they are recording.
   They just record based on the queue they are received on hence they don't know how to extract the actual timestamp from the messages.
   For example, there's actually an instance of each logger for each set of data being archived, so as well as Darwin they also record Trust, TD & ISS Telemetry.

[petermount]

Just as an addition to this, an incidence of #3 above happened this morning where some data was lost upstream:

What Is Impacted: Darwin

We are pleased to advise that the issue with Darwin missing data has now been resolved. Reporting Data has been restored and service has returned to normal.

Many apologies for any disruption and inconvenience caused.

Who Is Impacted: NRE Users

Incident Reference: INC0076644

Incident Start Date / Time: 31/03/24 - 09:13 hrs
Incident Resolution Date / Time: 31/03/24 - 10:39

Next Update Due: There will be no further update

[Paul Clark]

Thanks, this has given us some ideas on how to solve/detect this issue - we're going to start using the sequence numbers.

On Thursday 28 March 2024 at 08:20:30 UTC petermount wrote:

[Paul Clark]

We have done some more investigation and found that applying the selectors "MessageType='TS' OR MessageType = 'SC'" whilst creating the consumer is what seems to be responsible for the vast majority of the duplicates:

darwinSession.CreateConsumer(darwinTopic, "MessageType='TS' OR MessageType = 'SC'");

We two systems at the same time for about an hour and found the following:

System with filter

Duplicate messages received: 131693

Messages not received (compared to the other system)  : 17018

System without filter:

Duplicate messages received: 2812 

Messages not received (compared to the other system): 39 

These figures seem to point to a bug when using the filter - maybe only in the C# library.

From now on, we'll use it without the filter and filter it ourselves.