Darwin Release 4.4 - 12th July 2022 22:00

[RailAleFan]

Hi all,

Just received on the RDG info services mailing list notification of Darwin Release 4.5 on 12th July including a Push Port upgrade to v17 (v16 will continue to be supported as with previous upgrades).

The schema changes also affect the Darwin web services and the email indicates that these will go live on the industry user production endpoints from that date but no mention over that channel regarding NRDP access.

Does anybody know how this upgrade is likely to affect the NRDP - e.g. will OpenLDBWS / OpenLDBSVWS also be upgraded on that date and a v17 Push Port topic become available?

Sounds promising though as this upgrade is dedicated to changes to train loading information...

Cheers!

[Peter Hicks]

Hello

NRDP will still output v16 data until such time as RDG asks CACI to upgrade it to provide a v17 feed.

Remember when NRDP (formerly D3) had the v12 feed?  Considerable work was done to rebuild the platform for v16, and a lot of that work was to make NRDP less dependent on the data version.  It's one of the reasons some of the filtering was taken off.

I'm hoping RDG will ask CACI to upgrade NRDP to the v17 feed so we can all benefit from the data.  As for the OpenLDBWS and OpenLDBSVWS services - they will have a new WSDL (see https://staging.realtime.nationalrail.co.uk/OpenLDBWS/wsdl.aspx?ver=2021-11-01 for what it'll probably be), but you can still make SOAP requests using the old WSDL and they'll still work.

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/4560c814-a136-4e92-8a8b-bcd2f1604b69n%40googlegroups.com.

[RailAleFan]

Thanks Peter, looks good re: the web services!

I wasn't working in this space (at least with Push Port anyway) at the time of the upgrade to v16 so wasn't following what was involved. The filtering certainly has its use cases (including in my own Darwin archiver project) and if there is significant use of filtered queues obviously a big saving in the resources required to provide the service.

However an option for the clean transactional feed (where all messages related to an update e.g. schedule/schedule/assication/TS/TS are contained within a single Pport/uR envelope) on the NRDP would be great going forward and would benefit those consuming an unfiltered queue for the purposes of replicating the Darwin database in real time...

Cheers

[Rail Delivery Group]

An interesting point Re: filtering.

As you surmise, the breaking up of uR messages with multiple message types (such as schedule, TS and association) into individual uR messages is a direct consequence of supporting the filtering option. If the messages aren't broken down when filtering for a certain message type you'd get any uR messages with that message type, including those with other message types. So for example, if you filtered for "schedules only" you would also get TS, association, etc. messages if they happen to be in the same uR message.

With how NRDP currently works it was an all or nothing choice, either have filtering which only returned the requested data but at the cost of unfiltered consumers getting uR messages split into individual messages, or have filtering that would return any uR message with the desired type, including any other message types that happen to be in it. The latter was deemed to be more impactful, with consumers recieving data they aren't expecting or know how to deal with, and hence the former was chosen.

It'd be interesting to have views on the value and benefit of filtering opposed to the cost to unfiltered users of supporting it, for input into future developments of the open push port feed.

[Peter Hicks]

Peter

On Wed, 6 Jul 2022 at 13:39, 'Rail Delivery Group' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

 

It'd be interesting to have views on the value and benefit of filtering opposed to the cost to unfiltered users of supporting it, for input into future developments of the open push port feed.

 

If I remember correctly, the number of users with filtering enabled was small, and we decided it wasn't worth implementing filtering purely for them.  But if it were possible to re-implement filtering, it could be done some of these ways:

• By using a header and filtering in an ActiveMQ subscription to match only messages with specific TIPLOCs in the header.  This has the downside that every message will be larger and even if you're receiving everything, you'll still get the additional header with TIPLOCs
• Filtering client-side is possible right now - either by client software matching specific TIPLOCs in a message and processing based on those, or in future by having something like an intermediate ActiveMQ server running client-side doing XPath matching on messages
• Publishing messages to a per-TIPLOC topic might work (e.g. DARWIN.LOCATION.EUSTON, DARWIN.LOCATION.WATFDJ, DARWIN.LOCATION.MKNSCEN) and users could subscribe to DARWIN.* and receive all messages.  However, there would be duplication and extra work needed to remove identical messages

If it were up to me, I'd probably want to engage with the people who wanted filtering to discuss their use-case and see if there's some other solution that works for everyone.  Maybe that's an easily-configurable custom setup that needs human intervention, or maybe that's just some guidance on whether or not it really is sensible to consume the whole Push Port on a little Raspberry Pi.

It reminds me a bit of a discussion with TfL when they released the Countdown (bus) API a few years ago.  Somebody wanted to develop an app with a feature which the service didn't implement.  There was a debate as to whether TfL should build this function into the API, or whether the developer should run and maintain a server to do their customised feature.  I don't think we reached a conclusion, other than "You cannot expect everyone else to do the leg-work if you want to innovate".

Peter

[Rail Delivery Group]

Thanks Peter.

You're correct that the ability to filter the NRDP push port by TIPLOC was removed as part of the upgrade from v12 to v16 so only the ability to filter by message type remains ("using JMS Selectors on the MessageType header"- quoting from the wiki, please don't anyone ask me what that means, I assume those who need to know these type of things will know :-) )

The word from the team is that yes, they are looking at updating NRDP to v17. One challenge, as with the migration from v12 to v16, will be how to handle the migration. I'm also told that v17 also contains some other changes to the schema in addition to the train loading information, to cater for updates due in later Darwin releases. That's all being considered and any upgrade will be subject to cost and won't happen immediately, but it can be hoped that v17 will be made available to open users.

I can also confirm that you're right, users of OpenLDB and OpenLDBSV will be able to access the latest versions of these services when they are released and that existing versions are not being withdrawn (at least at this time, they may be in the future but notification will be sent to any users if that happens, with time allowed for them to migrate to a supported version before the old version is switched off).

[RailAleFan]

That's interesting - I wasn't aware that filtering by location was previously supported on the NRDP version of Push Port.  I know it's a feature of the upstream system and it certainly makes sense in that space but much less so IMO in the open data environment and sounds like a significant overhead to implement for what sounds like not a huge demand.

In terms of message type filtering by JMS selector, RDG/CACI will know how much this is utilised by NRDP users and as noted above there are clear use cases and resource benefits for this.

But regarding the cost to NRDP Push Port consumers only interested in the unfiltered feed for the purposes of replicating the Darwin database as an alternative to going into the web services commercial tier, recreating Darwin transactions from the de-constructed feed is straight forward enough it's just that it is not an exact computer science and requires supervisor code to handle the reconstruction logic failing.

No big deal, it's just a bit "ugly" from a software / data engineering point of view!

Cheers

On Wednesday, July 6, 2022 at 2:07:25 PM UTC+1 Peter Hicks wrote:

Peter

[Malcolm Bovey]

Volumes on the Darwin feed suddenly dropped to zero just after 10pm tonight - which I'm guessing is not a coincidence...

[Evelyn Snow]

Hello

You're correct, this isn't a coincidence, this is a planned outage. I've
checked the emails NRE have sent about this, and I'll note that they were all
received on a RDG-specific address, and not via the address my push port
account is registered under, so if you've not received any direct communication
about this, this might be the reason why.

Evelyn

2022-07-12T15:19:38-0700 Malcolm Bovey <malcol...@gmail.com>:

> * By using a header and filtering in an ActiveMQ subscription to

> match only messages with specific TIPLOCs in the header.  This
> has the downside that every message will be larger and even if
> you're receiving everything, you'll still get the additional
> header with TIPLOCs

> * Filtering client-side is possible right now - either by client

> software matching specific TIPLOCs in a message and processing
> based on those, or in future by having something like an
> intermediate ActiveMQ server running client-side doing XPath
> matching on messages

> * Publishing messages to a per-TIPLOC topic might work (e.g.

> DARWIN.LOCATION.EUSTON, DARWIN.LOCATION.WATFDJ,
> DARWIN.LOCATION.MKNSCEN) and users could subscribe to DARWIN.* and
> receive all messages.  However, there would be duplication and
> extra work needed to remove identical messages
> If it were up to me, I'd probably want to engage with the people who
> wanted filtering to discuss their use-case and see if there's some
> other solution that works for everyone.  Maybe that's an
> easily-configurable custom setup that needs human intervention, or
> maybe that's just some guidance on whether or not it really is
> sensible to consume the whole Push Port on a little Raspberry Pi.
> It reminds me a bit of a discussion with TfL when they released the
> Countdown (bus) API a few years ago.  Somebody wanted to develop an
> app with a feature which the service didn't implement.  There was a
> debate as to whether TfL should build this function into the API, or
> whether the developer should run and maintain a server to do their
> customised feature.  I don't think we reached a conclusion, other
> than "You cannot expect everyone else to do the leg-work if you want
> to innovate".
> Peter
>

> --
> You received this message because you are subscribed to the Google Groups
> "A gathering place for the Open Rail Data community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to openraildata-t...@googlegroups.com.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/openraildata-talk/e9ba1b00-4c48-47b5-bda1-91f2bc0558ccn%40googlegroups.com.