Fwd: Rail Data Marketplace November Edition

[Matthew Burdett]

See below latest newsletter for updates on data wants.

---------- Forwarded message ---------
From: <raildatam...@raildeliverygroup.com>
Date: Mon, 4 Nov 2024, 16:59
Subject: Rail Data Marketplace November Edition
To: <matthewbu...@gmail.com>

November 2024 Rail Data Marketplace Welcome to your round up of news related to the Rail Data Marketplace (RDM). If at any time you wish to unsubscribe, email us at raildatam...@raildeliverygroup.com.   In this edition RDM highlights Network Rail update on ‘data wants’ Blogs Featured products New data products on the RDM RDM highlights It is hard to believe that RDM is celebrating its first birthday. Just one year ago, RDM was launch by the then Rail Minister, Huw Merriman at the RIA conference. One year later and RDM has close to 500 organisations and over 1,000 individuals who are registered on the platform. It makes available 150 data products, many of which are exclusively published on RDM. Perhaps the real number to celebrate is the 4,000 data sharing agreements that RDM has created without a lawyer once being involved. While that might be bad news for the legal profession, it is allowing an ease and speed of access to data which is unique and paves the way for the industry to realise the potential of the data which we hold to support innovation and improve the passenger experience. The entire industry should be proud of the success of RDM as it was truly a collaborative effort; the Department for Transport, Rail Supply Group, Network Rail, RSSB, GBRTT, RIA as well as colleagues across RDG and the Train Operating Companies have all contributed. Our thanks to everyone.    In this newsletter, that whole-industry theme is reinforced with contributions from Network Rail who describe how they make decisions about data sharing as well as a couple of SMEs who have just published data products which demonstrate that innovation is truly alive in our industry.    RDM continues to evolve and this month has been no exception. Recognising the importance of understanding the provenance of data, RSSB commissioned a report titled ‘Identifying metadata to support confident use of data in a data-driven railway’ which was published in December of last year. RDM has now implemented the recommendations of that report in full, allowing publishers to use the metadata framework described in the report to input information into RDM and allowing consumers to benefit from a greater understanding of each data product.     This really puts our industry at the forefront of data sharing practice across the UK and globally. We now have widespread adoption by both public and private sector organisations of a single secure platform which uses a common taxonomy, a consistent metadata structure, and a quick and robust mechanism for access to both open and chargeable data. As we approach the 200th anniversary of the railway, RDM is another demonstration of our industry’s continued commitment to using technology to connect people and places.    We hope that you enjoy this round up of news and for those attending the RIA conference, please come and say hello to us in the exhibitors area. Network Rail update on ‘Data Wants’ Network Rail is working to publish a minimum of 15 new data products on the Rail Data Marketplace by the end of the year. We are currently progressing with the publication of Level Crossing, Flood Delay Minutes, and Maintenance Activities. We are also responding to academic requests for sample LIDAR and CAD data. Additionally, we are in discussions to release GEMINI and TOPS data, which will further enhance the accessibility and utility of our rail data on the RDM. GEMINI data includes detailed information on train allocations, which is crucial for understanding the deployment and scheduling of rolling stock across the network. This data will provide valuable insights into train operations.   Furthermore, we are excited to announce ongoing work to open up the LINX service catalogue. This initiative will provide comprehensive access to a wide range of services and tools that support the management and operation of the rail network. Blogs A FRESH perspective: the new era in data-driven decision making  Thomas Gallagher, Data Policy Development Lead at Network Rail, talks about the FRESH Model, a data sharing assessment tool developed to provide an objective view as to when Network Rail should share data.  Improving the use of metadata on the Rail Data Marketplace and across the rail industry  Last year, RSSB commissioned research to ‘Identify metadata to support the confident use of data in a data-driven railway’. Revised metadata fields have now been added to RDM. Taken together with RDM’s existing metadata fields, this provides comprehensive coverage of the schema recommended by RSSB and should give valuable additional information to consumers.  Featured products Data Thistle API: UK wide live events data  Data Thistle hosts the UK's most comprehensive set of live events information - think gigs, theatre, festivals, sport, days out and more. This data has two primary uses to the rail industry. The first key use is inspiring travel. Advertising live events drives increased leisure travel through a range of platforms, including websites, apps, RTI platforms, social media and automated CRM feeds. The second use is business intelligence: live events provide a window into future demand, which rail businesses use to predict high passenger volumes. This allows them to make better commercial decisions around yield management, to set better levels of staffing operationally, and to plan engineering works to avoid high demand periods up to two years in advance. Introducing DataWharf's Fuzzy Station Search: making train travel more accessible  DataWharf's Fuzzy Station Search is an AI-driven solution designed to help users find the correct station name for their origin or destination, even when they may not know the exact station name or its unfamiliar spelling. Returns the Station Name, CRS code and postcode for close-match stations.   Darwin Real Time Train Information In response to feedback, Darwin Real Time Train Information data is now available in XML format. This pub/sub product provides predicted arrival times and reasons for train delays and cancellations. We’re aware the XML topic preview currently does not display data, but it is accessible using a Kafka client. Disruptions API The RDG Disruptions API is now available on RDM. This data product provides real-time updates on station messages, bulletins, disruptions and incident information.  List of new data products on the RDM Thank you for reading the latest edition of the Rail Data Marketplace newsletter. To read our previous newsletters please click here. If at any time you wish to unsubscribe, email us at raildatam...@raildeliverygroup.com.

[Evelyn Snow]

Happy to see I'm still a single digit percentage of "data sharing agreements", 2 to be exact!

Evelyn

On Mon, Nov 04, 2024 at 05:13:16PM +0000, Matthew Burdett wrote:
> See below latest newsletter for updates on data wants.
>
> ---------- Forwarded message ---------
> From: <raildatam...@raildeliverygroup.com>
> Date: Mon, 4 Nov 2024, 16:59
> Subject: Rail Data Marketplace November Edition
> To: <matthewbu...@gmail.com>
>
>
>

> [aff8909a-7d48-44b5-9690-4ac968ad6ee5]

> [7fa0fc19-4d3f-46af-ad80-ca220cb69af1]
> [145031bd-533c-49af-8538-3451a7c577ab]

>
> Thank you for reading the latest edition of the Rail Data Marketplace
> newsletter. To read our previous newsletters please click here. If at any time
> you wish to unsubscribe, email us at raildatam...@raildeliverygroup.com
> .  
>

> *

[Tom Cairns]

I have to say I find the comments about number of data agreements made quite interesting – it says nothing about the amount of data actually being pushed through the platform which may be an entirely different number entirely.

 

(I’m still at 0% of the data sharing agreements as I’m still against the anti-disparagement clause in the data consumer & publisher agreements, although there is a dialogue on it being changed.)

 

Tom

 

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/ZykCQFZlHRyaZWKc%40evelyn.moe.

[Peter Hicks (Poggs)]

On 4 Nov 2024, at 17:38, Tom Cairns <t...@swlines.co.uk> wrote:

I have to say I find the comments about number of data agreements made quite interesting – it says nothing about the amount of data actually being pushed through the platform which may be an entirely different number entirely.

I’m at the consumer end of a lot of those data sharing agreements.  A lot of them have only been downloaded once and left to gather dust because the data is in Excel format, or isn’t updated frequently enough, or in some cases, can’t be downloaded automatically.

I’m getting a bit frustrated with RDM actually, as the functionality to send updated datasets to an Amazon S3 bucket doesn’t appear to work at all.  It may be down to permissions, but there’s no way I can test it by sending over a sample file on demand, and I can’t get RDM Support to tell me which permissions an IAM user needs.  It shouldn’t be this difficult.  I’m not downloading data manually.

The news about GEMINI is positive, and maybe widespread publication of that set of data will give TOCs an incentive to make sure it’s actually correct.  LINX is also huge, as strategically that is the system we should use in place of TRUST.

I remain on standby for the rail industry to come to me and say “Can you help us make sure this stuff lands positively?” - but so far, nobody wants to pay for my time and I have bills to pay.  I hope that will change soon.

Peter

[Matthew Burdett]

Agree with tom, I'm subscribed to about 20 out of curiosity but only using 1 actively (train movements, to see how long my Kafka works until it breaks (so far 150,000 activations recorded and counting))

I have no use for the other stuff like planned train formations, what am I supposed to use it for other than reporting.

 The comments about 15 new data products by the end of the year sounds rather optimistic, I'm hoping that includes wheel impact detection that was trial. The users to active subscriptions ratio should close in a bit when the products become more interesting to play with. 

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/9591DBC2-6D89-4108-8229-AC3BF27CA06E%40poggs.co.uk.

[Adam Williams]

> I’m getting a bit frustrated with RDM actually, as the functionality to send updated datasets to an Amazon S3 bucket doesn’t appear to work at all.  It may be down to permissions, but there’s no way I can test it by sending over a sample file on demand, and I can’t get RDM Support to tell me which permissions an IAM user needs.  It shouldn’t be this difficult.  I’m not downloading data manually.

We just wrote a scraper that pretends to be a browser to automatically download files. If there's interest, it could probably be open-sourced.

We needed to use one of the data-sets in production (RDM was the only place they were offered) and it took a very long time indeed for the auto-sync-to-S3/GCS functionality to ship, despite being raised internally with RDG repeatedly. We couldn't wait that long. And to be honest, the functionality of the rest of the platform doesn't really inspire much confidence in me to rely on it in prod.

You'd think a way of programmatically getting out the data would be the #1 feature you implement in a data marketplace before getting it out the door, really.

Adam

[Peter Hicks (Poggs)]

Hi Adam

On 9 Nov 2024, at 21:25, 'Adam Williams' via A gathering place for the Open Rail Data community <openrail...@googlegroups.com> wrote:

> I’m getting a bit frustrated with RDM actually, as the functionality to send updated datasets to an Amazon S3 bucket doesn’t appear to work at all.  It may be down to permissions, but there’s no way I can test it by sending over a sample file on demand, and I can’t get RDM Support to tell me which permissions an IAM user needs.  It shouldn’t be this difficult.  I’m not downloading data manually.

We just wrote a scraper that pretends to be a browser to automatically download files. If there's interest, it could probably be open-sourced.

Yes please!  If you’ve already written it, it’s good PR for you/your company, and it also highlights the fact that there’s no easy way to download files.

FWIW, I am still trying to set up a Teams call with the support team for them to tell me what permissions are needed for an IAM user.  I’ve sent them two email responses, they seem to have received neither :(

Peter

[Adam Williams]

> Yes please!

Sure, https://github.com/raileasyuk/rdm-scraper

> FWIW, I am still trying to set up a Teams call with the support team for them to tell me what permissions are needed for an IAM user.  I’ve sent them two email responses, they seem to have received neither :(

The RDM support people really seem to like calls. I reported a bug in the frontend, gave them the exact reason why it was happening (they didn't do any browser testing) and a suggested fix as well and they still want a Teams call with me to discuss it.

I had some problems replying to the support system by email, for what it's worth - it was only when I logged into the portal (https://raildata.org.uk/dashboard/supportRequests/myRequests) and replied there using the web UI that my comments got seen by the support team. I don't know if that's it working as intended, i.e. that all replies go to /dev/null or what, but it was what happened for me.

Adam