Archive of Passenger Train Allocation and Consist feed

[Peter Hicks]

All,

I've created a data product on RDM containing daily archives of the Passenger Train Allocation and Consist feed at https://raildata.org.uk/dashboard/dataProduct/P-fac2476d-e83c-4f71-b49d-2d14b9053ef2/overview.

Many people just want historical data and aren't able to keep a consumer running 24x7 to gather data, so this product will allow you to catch up with what you might have missed.

Thanks to everyone for your feedback on how data should be split - I've used the same format used elsewhere, i.e. UNIX timestamp followed by a '$' character, then the raw message.

Peter

[Marsh Lane]

Hi Peter,

Firstly, can I echo the other comments congratulating on getting the Allocation data in to the RDM, and your archive.  Are there any plans to expand the coverage further, such as to the FOCs?

The main reason for the message, was there an outage on the PTAC feed on 3rd October?  I note your archive file for that day is 0 Bytes, and that David Wheatley's 'Gemini Push Port' hasnt listed any files for the same day either.  The 4th October appears to have worked ok being a bit larger than usual at 67MB, but not large enough to have accommodated two-days worth of data?  I have yet to start playing with the XML live feed, hopefully next week, so I am unsure if the data was actually flowing through the live feed.

Cheers.

[David Wheatley]

Hi,

My feed archive broke due to a networking problem at Hetzner on 1 October. While the Darwin archiver recovered, the Gemini one did not for some reason; the code behind both archivers is identical from the Kafka reader standpoint.

From my logging, I can see 28 (exponentially backed-off) failed attempts to reconnect to RDM's feed from my Gemini archiver (due to the issue at Hetzner), followed by a successful connection attempt but with no message flow for an unknown reason. I've since added in a read timeout to reconnect to the feed if I don't receive any messages for 15 minutes.

I can't actually see any outage of the feed itself from internal metrics of another ingester I maintain, which received messages from the feed without issue throughout 3 October.

David

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/52d5527d-2c4b-4df9-bf2a-e93b2e53b720n%40googlegroups.com.

[Peter Hicks]

Hello

On Sunday, 5 October 2025 at 22:08, Marsh Lane <marsh...@outlook.com> wrote:

Firstly, can I echo the other comments congratulating on getting the Allocation data in to the RDM, and your archive. Are there any plans to expand the coverage further, such as to the FOCs?

Freight and engineering trains are a completely separate challenge, but there are some people (including me) working on it.

The main reason for the message, was there an outage on the PTAC feed on 3rd October? I note your archive file for that day is 0 Bytes, and that David Wheatley's 'Gemini Push Port' hasnt listed any files for the same day either. The 4th October appears to have worked ok being a bit larger than usual at 67MB, but not large enough to have accommodated two-days worth of data? I have yet to start playing with the XML live feed, hopefully next week, so I am unsure if the data was actually flowing through the live feed.

Further to David's reply, this appears to be more than one issue.  When the log for 3rd October was uploaded to RDM, it was received successfully but appeared as zero bytes - which I've fixed by re-uploading the file, and it now appears with the right size.  There were 48,887 messages for the 3rd, which compares to 46,724 on the 2nd and 47,811 the day before which looks broadly in line with what I'd expect.  On top of this, there was a relatively minor issue around 0846 where I lost connectivity to the source system, along with other feeds - but it appears to have reconnected soon after.

If you'll excuse the use of Excel for analysis, this chart shows the number of messages received on a particular second throughout the day on the 3rd - I can't see any gaps wide enough to indicate an outage, or peaks that would indicate a recovery from an outage:

(Others can replicate this by downloading the logs from the RDM for 3rd October and charting the UNIX timestamps on the x axis, and the number of messages with that timestamp on the Y axis).

There's nothing in logs here which indicates a failure of connectivity in to the RDM's Kafka hosted instance.  However, rather than just shrug my shoulders and say "Not my problem", I'll work on setting up a Kafka consumer to monitor the output from the RDM as seen by end users and add alerting on top of that.

Peter

[Marsh Lane]

Peter,

Thanks for your detailed response and confirmation of what happened, that useful, also thanks for reuploading the file.

Cheers

Richard

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/NEwCqnOdxsx02czlnTtvLSiOuuV2smBSpssZ8kHfgc2ZLOmMSNFOwXQECOQqJQ3BOzkiwcEc-B_HGjUvmoZkrAlLTXlfvvPYOokszelosdE%3D%40poggs.co.uk.