Documenting accessing Darwin Push Port via Kafka

[Marks Polakovs]

Hello folks,

Over the holidays I was experimenting with accessing the Push Port data via Kafka rather than the ActiveMQ/STOMP method, and figured it might be worth me writing up my findings to hopefully help others.

I've made a copy of the Open Rail Data Wiki page on the Push Port in my personal namespace where I've started hacking it to bits to explain Kafka. The page itself is here, and my changes from the current version are here.

Feedback and corrections welcome (either by email or by editing the wiki page directly!). Once it's a bit more fleshed out and reviewed I might merge it back to the mainspace page.

Best,

Marks

[Adam Williams]

Hi Marks,

Great job putting something together.

I think it may need a giant health warning about the mess with partitions/receiving messages in the wrong order, or did RDG finally sort that out?

https://groups.google.com/g/openraildata-talk/c/oth6M9vnsKU/m/rHtISTmyAwAJ

https://raildata.org.uk/community/category/topic/postDetail/94

[Peter Hicks]

On Friday, 9 January 2026 at 19:32, 'Adam Williams' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

I think it may need a giant health warning about the mess with partitions/receiving messages in the wrong order, or did RDG finally sort that out?

I'd go check, but it appears users no longer have the ability to DescribeConsumerGroups, DescribeCluster and DescribeConfigs so I can't use Kadeck, a useful UI for troubleshooting.

Checking with a Python script though:

==============================

Topic     : prod-1010-Darwin-Train-Information-Push-Port-IIII2_0-JSON

Partition : 0

Offset    : 318596102

Key       : b'{"messageID":"ID:liv1-dwnpp102-49747-638894340902160133-1:22:1:1:323040199"}'

Value     : <built-in method value of cimpl.Message object at 0x105137a40>

==============================

==============================

Topic     : prod-1010-Darwin-Train-Information-Push-Port-IIII2_0-JSON

Partition : 0

Offset    : 318596103

Key       : b'{"messageID":"ID:liv1-dwnpp102-49747-638894340902160133-1:22:1:1:323040201"}'

Value     : <built-in method value of cimpl.Message object at 0x1051b1240>

==============================

==============================

Topic     : prod-1010-Darwin-Train-Information-Push-Port-IIII2_0-JSON

Partition : 0

Offset    : 318596104

Key       : b'{"messageID":"ID:liv1-dwnpp102-49747-638894340902160133-1:22:1:1:323040198"}'

Value     : <built-in method value of cimpl.Message object at 0x105137a40>

==============================

If the keys are still the JMS messageID header, then the messages will occasionally be out-of-order.  In fact, even within a single partition, the messages don't appear to be in order - the final component of the messageID is the sequence number, and it jumps from -199 to -201, then back to -198 again.  Looking across both partitions, all the messages are there, so to quote Eric Morcambe, "I'm playing all the right notes, but not necessarily in the right order".

So the ordering message exists issue still.

Peter